[gitlab-org/gitlab] Queue pipelines/jobs when ci_active_jobs limit is exceeded instead of hard-failing the pipeline

### ROOT CAUSE The issue arises because GitLab's current behavior is to hard-fail a new pipeline when the `ci_active_jobs` limit is exceeded. This limit is applied at the project level and is designed to prevent excessive resource usage. However, for projects with high-concurrency CI workloads, this behavior can be disruptive and inefficient. ### CODE FIX To address this issue, we can implement a job-level throttling mechanism within the pipeline. This approach will allow new pipelines to be created and persisted normally, while jobs are held in a `waiting_for_resource` state when the project's active job count is at the limit. As existing active jobs complete and capacity frees up, held jobs will be automatically transitioned to `pending` and picked up by runners. Here is a high-level overview of the steps to implement this fix: 1. **Modify the pipeline creation logic**: Ensure that new pipelines are created and persisted normally, regardless of the active job count. 2. **Implement a job queue**: Create a queue to hold jobs that are waiting for resources. 3. **Update job state transitions**: When an active job completes, check if there are any jobs in the queue. If so, transition the oldest job in the queue to `pending` and assign it to a runner. 4. **Handle pipeline deferral**: If a pipeline is created but cannot start immediately due to the active job limit, place it in a `waiting` state. When the active job count drops below the limit, automatically start the oldest waiting pipeline. Here is a simplified code snippet to illustrate the concept:

ruby
class Pipeline
  def initialize
    @jobs = []
    @waiting_jobs = []
  end

  def add_job(job)
    if active_job_count < ci_active_jobs_limit
      job.state = 'pending'
      @jobs << job
    else
      job.state = 'waiting_for_resource'
      @waiting_jobs << job
    end
  end

  def complete_job(job)
    @jobs.delete(job)
    if @waiting_jobs.any?
      next_job = @waiting_jobs.shift
      next_job.state = 'pending'
      @jobs << next_job
    end
  end

  private

  def active_job_count
    @jobs.count { |job| job.state == 'pending' }
  end

  def ci_active_jobs_limit
    # Retrieve the instance-level ci_active_jobs limit
    Gitlab::CurrentSettings.ci_active_jobs_limit
  end
end

class Job
  attr_accessor :state
end


By implementing this job-level throttling mechanism, GitLab can handle high-concurrency CI workloads more gracefully, reducing the need for manual retries and improving the overall developer experience.