Using Local Disk

YepCode provides a temporary directory for storing files during process execution. This directory is accessible through the TMP_DATA_DIR environment variable and is perfect for storing temporary files, processing data, or creating intermediate files that need to be accessed during your process execution.

Note

File Size Limits: Be mindful of file sizes when writing to the temporary directory, as there may be storage limits depending on your YepCode plan. See our plans and limits page for more information.

JavaScript

In JavaScript, you can access the temporary directory using process.env.TMP_DATA_DIR. Here’s how to use it:

const fs = require('fs');
const path = require('path');

// Get the temporary directory path
const tmpDir = process.env.TMP_DATA_DIR;

// Create a file in the temporary directory
const filePath = path.join(tmpDir, 'my-temp-file.txt');
fs.writeFileSync(filePath, 'Hello from YepCode!');

// Read the file back
const content = fs.readFileSync(filePath, 'utf8');
console.log('File content:', content);

// List files in the temporary directory
const files = fs.readdirSync(tmpDir);
console.log('Files in temp directory:', files);

Example: Using Streams for Large Files

const fs = require('fs');
const path = require('path');

const tmpDir = process.env.TMP_DATA_DIR;

// Write a large file using streams
const filePath = path.join(tmpDir, 'large-file.txt');
const writeStream = fs.createWriteStream(filePath);

// Write data in chunks
for (let i = 0; i < 100; i++) {
  writeStream.write(`Data line ${i}\n`);
}

writeStream.end();
console.log('Large file created successfully');

// Read the file using streams
const readStream = fs.createReadStream(filePath, { encoding: 'utf8' });
let lineCount = 0;

readStream.on('data', (chunk) => {
  const lines = chunk.split('\n');
  lineCount += lines.length - 1;
});

readStream.on('end', () => {
  console.log(`File contains ${lineCount} lines`);
});

Python

In Python, you can access the temporary directory using os.environ.get('TMP_DATA_DIR'). Here’s how to use it:

import os

# Get the temporary directory path
tmp_dir = os.environ.get('TMP_DATA_DIR')

# Create a file in the temporary directory
file_path = os.path.join(tmp_dir, 'my-temp-file.txt')
with open(file_path, 'w') as f:
    f.write('Hello from YepCode!')

# Read the file back
with open(file_path, 'r') as f:
    content = f.read()
print('File content:', content)

# List files in the temporary directory
files = os.listdir(tmp_dir)
print('Files in temp directory:', files)

Example: Processing CSV Data

import os
import csv

# Write CSV data to temporary file
csv_data = (
  "name,age,city\n"
  "John,30,New York\n"
  "Jane,25,Los Angeles\n"
  "Bob,35,Chicago\n"
)

csv_path = os.path.join(os.environ.get('TMP_DATA_DIR'), 'users.csv')
with open(csv_path, 'w') as f:
    f.write(csv_data)

# Process the CSV file
data = []
with open(csv_path, 'r') as f:
    reader = csv.DictReader(f)
    for row in reader:
        data.append(row)

print('Processed data:', data)

Caution

Temporary Nature: Files stored in TMP_DATA_DIR are automatically cleaned up after process execution. Do not rely on these files persisting between executions.

Use Cases

• Data Processing: Store intermediate files during data transformation workflows
• File Format Conversion: Create temporary files for format conversion operations
• Logging: Write detailed logs that can be processed or uploaded elsewhere
• Caching: Store temporary cache files for repeated operations within the same execution
• File Upload Processing: Store uploaded files temporarily before processing or forwarding

Working with YepCode Storage

Local disk and YepCode Storage work excellently together for comprehensive file handling workflows. Here are common patterns:

Uploading Local Files to Storage

After creating files in the temporary directory, you can upload them to persistent storage:

JavaScript

const fs = require('fs');
const path = require('path');

// Create a file in local disk
const tmpDir = process.env.TMP_DATA_DIR;
const localFilePath = path.join(tmpDir, 'processed-data.csv');

// Generate some data and write to local file
const csvData = 'name,age,city\nJohn,30,New York\nJane,25,Los Angeles';
fs.writeFileSync(localFilePath, csvData);

// Upload to YepCode Storage for persistence
await yepcode.storage.upload('exports/processed-data.csv', fs.createReadStream(localFilePath));
console.log('File uploaded to storage successfully');

Python

import os

# Create a file in local disk
tmp_dir = os.environ.get('TMP_DATA_DIR')
local_file_path = os.path.join(tmp_dir, 'processed-data.csv')

# Generate some data and write to local file
csv_data = 'name,age,city\nJohn,30,New York\nJane,25,Los Angeles'
with open(local_file_path, 'w') as f:
    f.write(csv_data)

# Upload to YepCode Storage for persistence
with open(local_file_path, 'rb') as f:
    obj = yepcode.storage.upload('exports/processed-data.csv', f)
    print(f'File uploaded to storage: {obj.name}')

Downloading from Storage for Local Processing

When you need to process files that don’t work well with streams, download them to local disk:

JavaScript

const fs = require('fs');
const path = require('path');

// Download file from storage to local disk
const tmpDir = process.env.TMP_DATA_DIR;
const localFilePath = path.join(tmpDir, 'downloaded-file.json');

const stream = await yepcode.storage.download('data/input-file.json');
stream.pipe(fs.createWriteStream(localFilePath));

// Now process the file locally (e.g., with libraries that need file paths)
const content = fs.readFileSync(localFilePath, 'utf8');
const data = JSON.parse(content);

// Process the data...
const processedData = data.map(item => ({ ...item, processed: true }));

// Save processed result back to local disk
const outputPath = path.join(tmpDir, 'processed-output.json');
fs.writeFileSync(outputPath, JSON.stringify(processedData, null, 2));

// Upload processed result back to storage
await yepcode.storage.upload('results/processed-output.json', fs.createReadStream(outputPath));

Python

import os
import json

# Download file from storage to local disk
tmp_dir = os.environ.get('TMP_DATA_DIR')
local_file_path = os.path.join(tmp_dir, 'downloaded-file.json')

content = yepcode.storage.download('data/input-file.json')
with open(local_file_path, 'wb') as f:
    f.write(content)

# Now process the file locally (e.g., with libraries that need file paths)
with open(local_file_path, 'r') as f:
    data = json.load(f)

# Process the data...
processed_data = [{**item, 'processed': True} for item in data]

# Save processed result back to local disk
output_path = os.path.join(tmp_dir, 'processed-output.json')
with open(output_path, 'w') as f:
    json.dump(processed_data, f, indent=2)

# Upload processed result back to storage
with open(output_path, 'rb') as f:
    obj = yepcode.storage.upload('results/processed-output.json', f)
    print(f'Processed file uploaded: {obj.name}')

Best Practices for Local Disk + Storage Workflows

1. Use Local Disk for Processing: When you need to work with files that require specific file paths or libraries that don’t support streams
2. Use Storage for Persistence: Upload important results to storage for long-term access and sharing
3. Clean Up: Remember that local disk files are automatically cleaned up, but storage files persist until manually deleted
4. Error Handling: Always handle potential failures in both local disk operations and storage uploads/downloads

For more information about YepCode Storage capabilities, see the Storage documentation.