Working with python_wheel_task

[Enriquema]

I have the necessity to work with python_wheel_task, and for this I need to change the common.py class as follows:

def _provide_config(self):
        self.logger.info("Reading configuration from --conf-file job option")
        conf_file = self._get_conf_file()
        if not conf_file:
            self.logger.info(
                "No conf file was provided, setting configuration to empty dict."
                "Please override configuration in subclass init method"
            )
            return self._read_parameters()
        else:
            self.logger.info(f"Conf file was provided, reading configuration from {conf_file}")
            return self._read_config(conf_file)

    @staticmethod
    def _read_parameters() -> Dict[str, Any]:
        config = {}
        for param in sys.argv[1:]:
            start_pattern = "--"
            end_pattern = "="
            key = param[param.find(start_pattern)+len(start_pattern):param.find(end_pattern)]
            start_pattern = '--'+key+'='
            value = param[param.find(start_pattern)+len(start_pattern):]
            config[key]=value
        return config

With this code, I can create a deployment task as:

- task_key: "mi_task"
            job_cluster_key: "default"
            python_wheel_task:
              package_name: "my_package"
              entry_point: "default_entrypoint"

And my post works with the following code:

{
  "job_id": 1,
  "python_named_params": {
        "param1": "value1",
        "param2": "value2"
 }
}

So I suggest you to modify the common class to include this new way to read parameters because will be usefull for python_wheel_task. Of course my methods are not the best, so modify it as you need it.