How to handle null values in the PipelineContext?

[daniel-fink-de]

First, I think this project is awesome! I have been looking for something like that already for quite a while but the other solutions I have found so far just didn't feel compelling to me. I am still new to the library, but I can already see that it will surely fill my needs, thanks!

I was wondering whether there is a good way of handling possible null values in the PipelineContext. Say I have two steps, DataLoadingStep and DataProcessingStep, where the latter one requires data from the first one, e.g.,

public sealed class MyPipeline(IPipelineStepFactory pipelineStepFactory)
{
    public IPipeline<MyPipelineResult> Build()
    {
        var context = new MyPipelineContext();
        var pipeline = new PipelineBuilder<MyPipelineContext, MyPipelineResult>(pipelineStepFactory, context)
            .Add<DataLoadingStep>()
            .Add<DataProcessingStep>()
            .Build();
    }
}

public sealed class DataLoadingStep(IPipelineStep<MyPipelineContext> nextStep) : IPipelineStep<MyPipelineContext>
{
    public IPipelineStep<MyPipelineContext> NextStep { get; set; } = nextStep;

    public async ValueTask ExecuteAsync(MyPipelineContext context, CancellationToken cancellationToken)
    {
        // perform data loading
        context.Data = "loaded data";
        
        await NextStep.ExecuteAsync(context, cancellationToken);
    }
}

public sealed class DataProcessingStep(IPipelineStep<MyPipelineContext> nextStep) : IPipelineStep<MyPipelineContext>
{
    public IPipelineStep<MyPipelineContext> NextStep { get; set; } = nextStep;

    public async ValueTask ExecuteAsync(MyPipelineContext context, CancellationToken cancellationToken)
    {
        // perform data processing using context.Data
        context.Result = new MyPipelineResult(context.Data);
        
        await NextStep.ExecuteAsync(context, cancellationToken);
    }
}

public sealed record MyPipelineResult(string Data);

public sealed class MyPipelineContext : PipelineContext<MyPipelineResult>
{
    public string Data { get; set; } = null!;
    public MyPipelineResult Result { get; set; } = null!;
    
    public override MyPipelineResult GetPipelineResult()
    {
        return Result;
    }
}

With this solution, I need to be sure to properly set the Data property of the context, otherwise, I will get a NullReferenceException when accessing the value later.

However, it would be nice to get rid of this risk and to have compile-time checks to be sure that the values are properly set.
One solution I could think of is to use different PipelineContext types for the different steps, i.e., only requiring the values that I have at hand at a specific point in the pipeline. However, I haven't figured out if this is possible. Are there any known workarounds to this issue?

[mvSapphire]

Thank you for your feedback! It's fantastic to hear that you're finding the library compelling for your needs!

I hope I understand you right, you want to obtain only step-required data during step execution. Unfortunately, there is no way to do that. The context it's something that is shared between all pipeline steps.. you can try to do some workarounds to use different PipelineContext but this is gonna be a crutch cause the library is not designed to do so. However, you could give AddIf<T>(predicate) a try and see if it aligns with what you're aiming for!

public sealed class MyPipeline(IPipelineStepFactory pipelineStepFactory)
{
    private bool ContextContainsAllRequiredData(MyPipelineContext context) => !string.IsNullOrEmpry(context.Data);

    public IPipeline<MyPipelineResult> Build()
    {
        var context = new MyPipelineContext();
        var pipeline = new PipelineBuilder<MyPipelineContext, MyPipelineResult>(pipelineStepFactory, context)
            .Add<DataLoadingStep>()
            .AddIf<DataProcessingStep>(ContextContainsAllRequiredData)
            .Build();
    }
}

[daniel-fink-de]

Thank you very much for your answer! In the meantime, I did a little more digging in the library itself and I saw why using multiple PipelineContext is not easy to achieve:

PowerPipe/src/PowerPipe/Pipeline.cs

Lines 9 to 14 in 2b75ff3

	public class Pipeline<TContext, TResult> : IPipeline<TResult>
	where TContext : PipelineContext<TResult>
	where TResult : class
	{
	private readonly TContext _context;
	private readonly IPipelineStep<TContext> _initStep;

The Pipeline always has the next step as property. This means that StepA would need to know the type of PipelineContext of StepB, essentially a coupling to "the future". One could get rid of this if the Pipeline itself would do the orchestration of the execution (i.e., not a PipelineStep is executing the next step, but the Pipeline executes StepA, takes the response and puts it into StepB. I have created a minimal setup with this in mind: https://gist.github.com/daniel-fink-de/cdec40f4223c90f30b2dcaaff53eab16

However, I am using heavily reflection and I can't use dependency injection since I need to create the instances when calling the AddStep function:

var pipeline = new PipelineBuilder()
    .AddStep(new LoadDataStep())
    .AddStep(new ProcessDataStep())
    .Build();

var result = pipeline.Run(input);

return result;

I think it should be possible to get rid of these limitations by using source code generators. Have you thought about using this for the library? Similar like Mapster does it?

But anyway, this is out of the scope of the discussion. I think a quick workaround is either taking your suggestion of the AddIf, or introducing the "check" functionality in the context itself, with a "Result wrapper", something like e.g.,

public record SamplePipelineResult(int Value);

public class SamplePipelineContext : PipelineContext<Result<SamplePipelineResult>>
{
    private int? _test1;

    public Result<int> Test1
    { 
        get
        {
            if (_test1.HasValue)
            {
                return Result<int>.Success(_test1.Value);
            }
            else
            {
                return Result<int>.Failure("Test1 is not set");
            }
        }
        set
        {
            if (value.IsSuccess)
            {
                _test1 = value.Value;
            }
            else
            {
                _test1 = null;
            }
        }
    }

    public override Result<SamplePipelineResult> GetPipelineResult()
    {
        if (_test1.HasValue)
        {
            return Result<SamplePipelineResult>.Success(new SamplePipelineResult(_test1.Value));
        }
        else
        {
            return Result<SamplePipelineResult>.Failure("Test1 is not set");
        }
    };
}

[mvSapphire]

Thank you for sharing your insights and the code snippets!

I agree that decoupling the Pipeline and its steps offers more flexibility and scalability. Your suggestion of orchestrating the execution within the Pipeline itself rather than relying on each step to execute the next is an interesting approach.

Your idea of using source code generators sounds promising. However, I think that these changes could introduce a lot of breaking changes and require a lot of design/architecture changes. IMO this library solves the problems for which it was created.

Overall, your willingness to explore different approaches and suggest improvements is commendable. I'll consider your suggestions!