Freeing RowVector in Velox

[aarati-K]

I'm trying to optimize the memory consumption of my velox program as I'm running out of memory on my local machine. I have a few RowVectors in my code that are no longer used, and freeing them would enable me to run the program on my local machine. Below I describe the code snippet in detail.

The code below gradually generates a star schema starting from an initial wide table. In each iteration, I'm adding a key column to the current fact table generating the variable temp_table, which is then decomposed into a dimension table (using aggregation) and a new more concise fact table (projecting columns, and a window function over the key column). I have two questions:

1. In each iteration, a subset of columns from the temp_table are projected to the newly generated fact_table. Does Velox create a copy of these projected columns?
2. If Velox does create a copy, how would I go about freeing the temp_table since it is not used in the next iteration?

// Iterate over all the col groups to generate dimension tables and fact table
// We start initially with a wide _table
fact_table_local = _table;
auto group_no = 0;

// Fact table col names are updated as we update the fact table
vector<string> fact_table_col_names;
auto num_cols = fact_table_local->children().size();
for (auto i=0; i<num_cols; i++) {
	fact_table_col_names.push_back(string("c") + to_string(i));
}
for (auto dim_col_names : col_groups_names) {
	// Add a key column to the fact table to generate temp_table
	vector<int64_t> keys;
	auto num_rows = fact_table_local->size();
	for (auto i=0; i<num_rows; i++) {
		keys.push_back(i);
	}
	auto key_col = makeFlatVector<int64_t>(keys);
	vector<VectorPtr> cols;
	for (auto col : fact_table_local->children()) {
		cols.push_back(col);
	}
	cols.push_back(key_col);
	vector<string> temp_table_col_names(fact_table_col_names);
	temp_table_col_names.push_back("key");
        // temp_table created by adding a key column to the current fact table
	auto temp_table = makeRowVector(temp_table_col_names, cols);

	// Generate the dimension table
	string aggregate = "min(key) as p" + to_string(group_no);
	auto plan = PlanBuilder()
				.values({temp_table})
				.aggregation(
					dim_col_names,
					{aggregate},
					{},
					core::AggregationNode::Step::kPartial,
					false)
				.planNode();
	auto dim_table = AssertQueryBuilder(plan).copyResults(pool());

	...

	// Generate the fact table

	// Generate the window clause
	string window_clause = string("min(key) OVER (PARTITION BY ");
	auto j = 0;
	for (auto dim_col_name : dim_col_names) {
		if (j != 0) window_clause += string(", ");
		window_clause += dim_col_name;
		j += 1;
	}
	window_clause += " ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) as p" + to_string(group_no);

	// Generate the fact table column names to be projected out
	vector<string> project_col_names;
	for (auto col_name : fact_table_col_names) {
		if (find(dim_col_names.begin(), dim_col_names.end(), col_name) != dim_col_names.end())
			// A dimension column
			continue;
		project_col_names.push_back(col_name);
	}
	project_col_names.push_back(string("p") + to_string(group_no));
	fact_table_col_names = project_col_names;

	plan = PlanBuilder()
			.values({temp_table})
			.window({window_clause})
			.project(fact_table_col_names)
			.planNode();
	fact_table_local = AssertQueryBuilder(plan).copyResults(pool());

        //// How to free the temp_table? It is unused in the next iteration.

	group_no++;
}

Any inputs are appreciated!

[aarati-K]

Any help regarding this issue would be much appreciated!

[aditi-pandit]

@aarati-K :
temp_table.reset(nullptr) or just temp_table = nullptr should release the memory imo. The RowVectorPtr is just a regular shared_ptr. Once it goes out of scope, the memory would be released.