Merge pull request #156 from acep-uaf/nicole/updates

Last update/release before KOTZ

README.md

@@ -19,11 +19,11 @@ Ensure you have the following before running the pipeline:
 - FTP credentials for the meter
 - Meter Configuration
 - Installed on `camio-ot-dev`:
-    - `lftp` — FTP Operations
-    - `yq` - YAML Parsing
-    - `zip` - Compressing Data
-    - `rsync` — Transfering Data
-    - `jq` — JSON Parsing
+    - `lftp`
+    - `yq`
+    - `zip`
+    - `rsync`
+    - `jq`
 
 ## Installation
 1. You must be connected to the `camio-ot-dev` server. See **camio-ot-dev(SSH)** in the [ACEP Wiki](https://wiki.acep.uaf.edu/en/teams/data-ducts/aetb).
@@ -39,35 +39,23 @@ Ensure you have the following before running the pipeline:
 
 ## Configuration
 
-### Data Pipeline Configuration
-1. Navigate to the `config` directory and copy the `config.yml.example` file to a new `config.yml` file:
+### General Configuration Steps
+1. Navigate to the `config` directory and copy the example configuration file to a new file:
 
     ```bash
     cd config
     cp config.yml.example config.yml
-    ```
-
-2. **Update** the `config.yml` file with the FTP credentials and meter configuration data.
-
-3. Secure the `config.yml` file so that only the owner can read and write:
-
-    ```bash
-    chmod 600 config.yml
-    ```
-
-### Archive Pipeline Configuration
-1. Navigate to the `config` directory and copy the `archive_config.yml.example` file to a new `archive_config.yml` file:
-
-    ```bash
-    cd config
     cp archive_config.yml.example archive_config.yml
     ```
 
-2. **Update** the `archive_config.yml` file with the source and destination directories and details.
+2. **Update** the configuration files with the necessary details:
+    - **`config.yml`**: Add the FTP server credentials and meter configuration data.
+    - **`archive_config.yml`**: Add the source and destination directories and other relevant details.
 
-3. Secure the `archive_config.yml` file so that only the owner can read and write:
+3. Secure the configuration files so that only the owner can read and write:
 
     ```bash
+    chmod 600 config.yml
     chmod 600 archive_config.yml
     ```
 
@@ -81,7 +69,7 @@ To run the data pipeline and then transfer data to the Data Acquisition System (
     ### Command
 
     ```bash
-    ./data_pipeline.sh -c /path/to/config.yml
+    ./data_pipeline.sh -c config/config.yml
     ```
 
 2. **Run the Archive Pipeline**
@@ -91,50 +79,59 @@ To run the data pipeline and then transfer data to the Data Acquisition System (
     ### Command
 
     ```bash
-    ./archive_pipeline.sh -c /path/to/archive_config.yml
+    ./archive_pipeline.sh -c config/archive_config.yml
     ```
     #### Notes
     The **rsync** uses the `--exclude` flag to exclude the `working` directory to ensure only complete files are transfered. 
 
 3. **Run the Cleanup Process (Conditional)**
 
-    If the `archive_pipeline` script completes successfully and the `enable_cleanup` flag is set to true in the archive configuration file, the `cleanup.sh` script will be executed automatically. This script removes outdated event files from level0 based on the retention period specified in the configuration file.
+    If the `archive_pipeline` script completes successfully and the `enable_cleanup` flag is set to true in the archive configuration file, the `cleanup.sh` script will be executed automatically. This script removes outdated event files from `level0` based on the retention period specified in the configuration file.
 
-    #### Notes
-    Ensure that the `cleanup.sh` script is configured correctly in the `archive_config.yml` file to specify the retention period for each directory set for the cleanup process.
+    If the `enable_cleanup` flag is not enabled, you can run the cleanup manually by passing in the archive configuration file.
+
+    ### Command
 
+    ```bash
+    ./cleanup.sh -c config/archive_config.yml
+    ```
+
+    #### Notes
+    Ensure that the `archive_config.yml` file is properly configured with the retention periods for each directory in the cleanup process.
 
 ## How to Stop the Pipeline
 
 When you need to stop the pipeline:
 
 - **To Stop Safely/Pause Download**: 
   - Use `Ctrl+C` to interrupt the process. 
+  - If interupting the proccess doesn't work try `Ctrl+\` to quit.
   - If you would like to resume the download, rerun the `data_pipeline`command.The download will resume from where it left off, provided the same config file (`-c`)is used.
 - **Avoid Using `Ctrl+Z`**: 
   - **Do not** use `Ctrl+Z` to suspend the process, as it may cause the pipeline to end without properly closing the FTP connection.
 
+## Testing (IN PROGRESS)
 
-## Testing
+This repository includes automated tests for the scripts using [Bats (Bash Automated Testing System)](https://github.com/bats-core/bats-core) along with helper libraries: `bats-assert`, `bats-mock`, and `bats-support`. The tests are located in the `test` directory.
 
-This repository includes automated tests for the scripts using [Bats (Bash Automated Testing System)](https://github.com/bats-core/bats-core). The tests are located in the `test` directory. 
-
-[Bats Documentation](https://bats-core.readthedocs.io/en/stable/index.html)
 ### Prerequisites
 
-Ensure you have `bats-core` installed. You can install it using the following steps:
+Ensure you have cloned the repository with its required submodules, they should be located under the `test` and `test/test_helper` directories:
+- `bats-core`
+- `bats-assert`
+- `bats-mock`
+- `bats-support`
 
-1. **On Ubuntu/Debian**:
+1. **Clone the repository with submodules**:
 
     ```bash
-    sudo apt-get update
-    sudo apt-get install -y bats
+    git clone --recurse-submodules git@github.com:acep-uaf/camio-meter-streams.git
     ```
 
-2. **On macOS (using Homebrew)**:
+    If you have already cloned the repository without submodules, you can initialize and update them with:
 
     ```bash
-    brew install bats-core
+    git submodule update --init --recursive
     ```
 
 ### Running the Tests
@@ -148,12 +145,5 @@ Ensure you have `bats-core` installed. You can install it using the following st
 2. **Run all the tests**:
 
     ```bash
-    bats test/
-    ```
-
-3. **Run specific test files**:
-
-    ```bash
-    bats test/test_data_pipeline.bats
-    bats test/test_commons.bats
-    ```
+    bats test
+    ```

cli_meter/common_utils.sh

@@ -13,7 +13,8 @@
 #   exlock()                - Obtain an exclusive lock
 #   shlock()                - Obtain a shared lock
 #   unlock()                - Drop a lock
-#   fail()                  - Output an error message and exit
+#   failure()               - Output an error message and exit
+#   warning()               - Output a warning message
 #   log()                   - Output a message to stderr
 #   show_help_flag()        - Display usage information
 #
@@ -76,6 +77,11 @@ failure() {
   exit $exit_code
 }
 
+warning(){
+  local message="${1:-""}"
+  log "[WARNING] $message"
+}
+
 log() {
   echo "$1" >&2
 }
@@ -124,6 +130,7 @@ show_help_flag() {
 # Export functions for use in other scripts
 export -f log
 export -f failure
+export -f warning
 export -f show_help_flag
 
 export -f _lock

cli_meter/data_pipeline.sh

@@ -5,11 +5,10 @@
 #                     loops through the meters listed in the configuration file
 #                     and downloads event files using the appropriate credentials.
 #
-# Usage:              ./data_pipeline.sh -c <config_path> [-d <download_dir>]
+# Usage:              ./data_pipeline.sh -c <config_path>
 #
 # Arguments:
 #   -c, --config       Path to the configuration file
-#   -d, --download_dir Optional: Override the download directory from the config
 #   -h, --help         Show usage information
 #
 # Requirements:       yq
@@ -74,7 +73,8 @@ for ((i = 0; i < num_meters; i++)); do
     if "$current_dir/meters/$meter_type/download.sh" "$meter_ip" "$output_dir" "$meter_id" "$meter_type" "$bandwidth_limit" "$data_type" "$location"; then
         log "Download complete for meter: $meter_id"
     else
-        # This will be changed to a warning in the future
-        failure $STREAMS_DOWNLOAD_FAIL "Download incomplete for meter: $meter_id"
+        warning "Download incomplete for meter: $meter_id"
     fi
 done
+
+log "All meters have been processed"

cli_meter/meters/sel735/cleanup_incomplete.sh

@@ -2,7 +2,7 @@
 # ==============================================================================
 # Script Name:        cleanup_incomplete.sh
 # Description:        This script deletes all directories matching the pattern
-#                     *.incomplete_<digit> within a given directory.
+#                     *.incomplete_<digit> within a given directory and all empty directories.
 #
 # Usage:              ./cleanup_incomplete.sh <directory>
 #
@@ -30,4 +30,9 @@ find "$base_directory" -type d -regex '.*/.*\.incomplete_[0-9]+' -print0 | while
   rm -rf "$dir" && log "Successfully deleted directory: $dir" || log "Failed to delete directory: $dir"
 done
 
+# Find and delete all empty directories
+find "$base_directory" -type d -empty -print0 | while IFS= read -r -d '' dir; do
+  rmdir "$dir" && log "Successfully deleted empty directory: $dir" || log "Failed to delete empty directory: $dir"
+done
+
 log "Cleanup process completed in directory: $base_directory"

cli_meter/meters/sel735/common_sel735.sh

@@ -20,7 +20,6 @@ handle_sig() {
     if [ -n "$current_event_id" ]; then
         local output_dir="$base_output_dir/$date_dir/$meter_id"
         mark_event_incomplete "$current_event_id" "$output_dir"
-        log "Download in progress, moving event $current_event_id to .incomplete"
     else
         log "No download in progress, no event to move to .incomplete"
     fi
@@ -106,7 +105,11 @@ validate_complete_directory() {
     # Check for metadata files
     local metadata_files=("${event_id}_metadata.yml" "checksum.md5")
     for file in "${metadata_files[@]}"; do
-        [ ! -f "${event_dir}/${file}" ] && log "Missing metadata file: ${file} in directory: ${event_dir}" && return 1 # False - Metadata file is missing
+        log "Checking for metadata file: ${file} in directory: ${event_dir}"
+        if [ ! -f "${event_dir}/${file}" ]; then
+            log "Missing metadata file: ${file} in directory: ${event_dir}"
+            mark_event_incomplete "$event_id" "$(dirname "$event_dir")" && return 1 # False - Metadata file is missing
+        fi
     done
 
     return 0 # True - All files are present

cli_meter/meters/sel735/create_message.sh

@@ -39,5 +39,5 @@ json_payload=$(jq -n \
     '{id: $id, filename: $fn, path: $pth, data_type: $dt}')
 
 # Write the JSON payload to the .message file
-echo "$json_payload" > "$message_file" && log "Created message file: $message_file" || failure $STREAMS_FILE_CREATION_FAIL "Failed to write to message file: $message_file"
+echo "$json_payload" > "$message_file" && log "Created message file: $message_file" || warning "Failed to write to message file: $message_file"
 

cli_meter/meters/sel735/download.sh

@@ -42,6 +42,8 @@ bandwidth_limit="$5"
 data_type="$6"
 location="$7"
 
+log "Starting download process for meter: $meter_id"
+
 # Make dir if it doesn't exist
 mkdir -p "$base_output_dir"
 
@@ -97,7 +99,7 @@ for event_info in $("$current_dir/get_events.sh" "$meter_ip" "$meter_id" "$base_
   # Execute create_message.sh
   "$current_dir/create_message.sh" "$event_id" "$zip_filename" "$path" "$data_type" "$event_zipped_output_dir" || {
     mark_event_incomplete
-    failure $STREAMS_FILE_CREATION_FAIL "Failed to create message file"
+    warning $STREAMS_FILE_CREATION_FAIL "Failed to create message file"
   }
 
 done

cli_meter/test/test_commons.bats

@@ -55,7 +55,6 @@ teardown() {
   base_output_dir="$TMP_DIR"
   run handle_sig SIGINT
   assert_failure 130
-  assert_output --partial "Download in progress, moving event $EVENT_ID to .incomplete"
   assert_output --partial "130"
 }