<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[KivaroTools Engineering]]></title><description><![CDATA[Explore practical insights on web development, software engineering, browser-based tools, application security, performance optimization, and financial computing from KivaroTools.]]></description><link>https://kivarotools.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6ae39971306486e1bf4be068/8a1b61fd-335c-46cc-b132-16ca6ba7f2e4.png</url><title>KivaroTools Engineering</title><link>https://kivarotools.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Sat, 07 Nov 2026 11:25:43 GMT</lastBuildDate><atom:link href="https://kivarotools.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[How to Read Parquet, Avro and Feather Files in the Browser with TypeScript: A 5-Stage Pipeline Walkthrough]]></title><description><![CDATA[Explore a 5-stage TypeScript pipeline to parse Parquet, Avro, and Feather files in the browser using Web Workers, Thrift footers, Snappy/LZ4, and BigInt.]]></description><link>https://kivarotools.hashnode.dev/how-to-read-parquet-avro-and-feather-files-in-the-browser-with-typescript-a-5-stage-pipeline-walkthrough</link><guid isPermaLink="true">https://kivarotools.hashnode.dev/how-to-read-parquet-avro-and-feather-files-in-the-browser-with-typescript-a-5-stage-pipeline-walkthrough</guid><category><![CDATA[data-engineering]]></category><category><![CDATA[JavaScript]]></category><category><![CDATA[Apache Parquet]]></category><category><![CDATA[BigInt]]></category><category><![CDATA[data-quality]]></category><category><![CDATA[TypeScript]]></category><category><![CDATA[web workers]]></category><category><![CDATA[apache arrow]]></category><dc:creator><![CDATA[Muhammad Ahmad]]></dc:creator><pubDate>Wed, 04 Nov 2026 03:57:50 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ae39971306486e1bf4be068/a2d75d97-25d5-4c26-b8e4-68cd52bb13c3.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Building an in-browser Parquet viewer: magic bytes, Thrift footers, ZigZag varints, Snappy and LZ4 &amp; LZ4_RAW decoding, BigInt precision and Web Workers.</p>
<p>A browser tab has no <code>pyarrow</code>, no <code>fastavro</code> and no native libraries. To open a Parquet, Avro or Arrow file there, you read the bytes yourself, and you do it without freezing the page or damaging 64-bit integers.</p>
<p>That is the problem behind the KivaroTools Parquet, Avro &amp; Feather viewer. Its engine, <code>lib/columnarViewerEngine.ts</code>, runs in a background Web Worker (with a main-thread fallback) through a five-stage pipeline. This post walks through the stages. Each one explains a specific reason why these files sometimes refuse to open, so it is useful even if you never write a reader.</p>
<p><em>New to the formats?</em> Parquet is a columnar format for analytics, Avro is a row-oriented format for moving records together with their schema, and Feather is an Arrow-based format for passing dataframes between tools.</p>
<h2>Stage 1: Signature and tail inspection</h2>
<p>Extensions lie. Object stores hand you <code>part-00000.snappy.parquet</code> or nothing at all, and Arrow files get renamed between <code>.feather</code>, <code>.arrow</code> and <code>.ipc</code>. So the first job is to read the leading and trailing bytes of the buffer:</p>
<table>
<thead>
<tr>
<th>Bytes</th>
<th>Format</th>
</tr>
</thead>
<tbody><tr>
<td><code>PAR1</code> at head <strong>and</strong> tail</td>
<td>Apache Parquet</td>
</tr>
<tr>
<td><code>Obj</code> + <code>0x01</code></td>
<td>Avro Object Container File</td>
</tr>
<tr>
<td><code>FEA1</code></td>
<td>Feather V1</td>
</tr>
<tr>
<td><code>ARROW1</code></td>
<td>Feather V2 / Arrow IPC file</td>
</tr>
<tr>
<td><code>0xFFFFFFFF</code></td>
<td>Arrow IPC stream</td>
</tr>
<tr>
<td><code>PARE</code></td>
<td>Parquet modular encryption</td>
</tr>
</tbody></table>
<p>A simplified sketch of the idea (not the engine's source):</p>
<pre><code class="language-ts">function sniff(b: Uint8Array): string {
  const ascii = (i: number, n: number) =&gt; String.fromCharCode(...b.slice(i, i + n));
  if (ascii(0, 4) === "PARE") return "parquet-encrypted";
  if (ascii(0, 4) === "PAR1" &amp;&amp; ascii(b.length - 4, 4) === "PAR1") return "parquet";
  if (b[0] === 0x4f &amp;&amp; b[1] === 0x62 &amp;&amp; b[2] === 0x6a &amp;&amp; b[3] === 0x01) return "avro-ocf";
  if (ascii(0, 4) === "FEA1") return "feather-v1";
  if (ascii(0, 6) === "ARROW1") return "feather-v2";
  if (b[0] === 0xff &amp;&amp; b[1] === 0xff &amp;&amp; b[2] === 0xff &amp;&amp; b[3] === 0xff) return "arrow-ipc-stream";
  return "unknown";
}
</code></pre>
<h2>Stage 2: Footer and schema extraction</h2>
<p><strong>Why Parquet keeps its schema at the end.</strong> Engines such as Spark, Trino and DuckDB stream data into row groups and column chunks without knowing the final row count, byte offsets or min/max statistics. Only after the last row group is flushed can the writer append the <code>FileMetaData</code> structure, then its 4-byte length, then the closing <code>PAR1</code>. So a reader always starts with the last 8 bytes:</p>
<pre><code class="language-ts">const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
const footerLen = view.getUint32(bytes.length - 8, true); // little-endian
const footerStart = bytes.length - 8 - footerLen;
</code></pre>
<p>A download that stops early can begin with a perfectly valid <code>PAR1</code> and still be unreadable, because the tail is what proves the file is complete.</p>
<p>The footer is Thrift Compact Protocol. The engine implements its own <code>ThriftCompactReader</code> over <code>Uint8Array</code> and <code>DataView</code>: field IDs are stored as deltas, types are small codes, and structures nest as structs inside lists. That is enough to rebuild the schema, row-group and column-chunk metadata, offsets, encodings and statistics.</p>
<p><strong>Why Avro does the opposite.</strong> An Avro Object Container File is built for append-heavy event streams. Right after the 4-byte magic it carries a header map with the full JSON schema (<code>avro.schema</code>) and the codec (<code>avro.codec</code>), followed by a random 16-byte sync marker that separates data blocks. Integers and string lengths use ZigZag varints, which map signed values to unsigned ones so small numbers of either sign take one byte:</p>
<table>
<thead>
<tr>
<th>Signed value</th>
<th>0</th>
<th>-1</th>
<th>1</th>
<th>-2</th>
<th>2</th>
</tr>
</thead>
<tbody><tr>
<td>ZigZag</td>
<td>0</td>
<td>1</td>
<td>2</td>
<td>3</td>
<td>4</td>
</tr>
</tbody></table>
<p>For <code>long</code> values that can use all 64 bits, the decode is a neat BigInt exercise:</p>
<pre><code class="language-ts">function zigzagDecode(n: bigint): bigint {
  return (n &gt;&gt; 1n) ^ -(n &amp; 1n);
}
</code></pre>
<h2>Stage 3: Page decompression and RLE decoding</h2>
<p>Column chunks and data blocks are decompressed with pure TypeScript decoders: Snappy (an LZ77-style state machine of literals and copy offsets), GZIP/DEFLATE, and LZ4. Then dictionary indices, repetition levels, definition levels and null bitmaps are rebuilt with the Hybrid RLE / Bit-Packing routine.</p>
<p>Two details are worth knowing:</p>
<ul>
<li><strong>"LZ4" is not one wrapper.</strong> A Parquet column labelled LZ4 can hold a standard LZ4 frame (magic <code>0x184D2204</code>), Hadoop/Parquet-style eight-byte big-endian framing, or a raw block of literals and match offsets. A decompressor that assumes one fails on the others.</li>
<li><strong>Treat compressed input as untrusted.</strong> The RFC 1951 inflate path validates Huffman code lengths, rejects oversubscribed trees with a Kraft-inequality check, verifies stored-block <code>LEN</code>/<code>NLEN</code>, and enforces a 250 MB limit on decompressed page size. A small payload can expand dramatically, and a tab has finite memory.</li>
</ul>
<h2>Stage 4: BigInt guard and column profiling</h2>
<p>Decoding an <code>INT64</code> correctly is only half the job. Supported 64-bit values are read with <code>getBigInt64()</code> / <code>getBigUint64()</code> (or BigInt varint arithmetic for Avro), and values outside ±9,007,199,254,740,991 stay exact strings across display, sorting and filtering. Numeric comparisons go through <code>compareExactNumericValues()</code>, which compares BigInt values and high-precision decimal strings without converting to floats.</p>
<p>The profiler follows the same split. It computes null ratios and distinct cardinality for every column, exact BigInt min/max for unsafe integer columns, and mean, standard deviation and sample P25/P50/P75 only for ordinary IEEE-754 columns. Float-based quantiles are never presented as exact BigInt statistics.</p>
<h2>Stage 5: Query, DDL and safe serialization</h2>
<p>Everything after decoding runs in memory over the decoded rows: filtering with 11 operators (Contains, =, ≠, &gt;, ≥, &lt;, ≤, Starts With, Ends With, Is NULL, Is Not NULL) combined with AND or OR, NULL-last sorting, and column projection. Results can be serialized to 10 schema or SDK targets (PostgreSQL, Snowflake, BigQuery, DuckDB, ClickHouse, TypeScript, PyArrow, Polars, Avro and JSON Schema) or to export formats including CSV, native <code>.xlsx</code>, TSV, JSON, JSONL, SQL INSERT, Markdown and HTML, with a formula-injection guard for spreadsheet output.</p>
<p>One boundary worth stating plainly: filtering does not use Parquet row-group statistics. The Binary X-Ray tab displays row-group zone maps for inspection, but the grid does not skip row groups the way a query engine would.</p>
<h2>The worker around the pipeline</h2>
<p>Parsing a large file on the main thread freezes the UI, so <code>createColumnarParseWorkerDispatcher()</code> runs the engine in a worker built at runtime by <code>buildColumnarWorkerScriptSource()</code>. It uses monotonic request IDs and cancellation, so if someone drops a large file and then a small one, the stale result cannot overwrite the new one: the old worker is terminated, its Blob URL revoked and its promise rejected. Input bytes are cloned before transfer so the caller's buffer survives, and if worker setup fails it falls back to main-thread parsing, trading UI responsiveness for compatibility.</p>
<h2>What it does not decode</h2>
<p>A reader that prints plausible rows when it cannot decode something is worse than one that says no. The engine reports these as unsupported:</p>
<ul>
<li><strong>Parquet:</strong> ZSTD, BROTLI, LZO; DELTA_BINARY_PACKED; BYTE_STREAM_SPLIT; multi-level Dremel repetition; KMS-based modular encryption.</li>
<li><strong>Avro:</strong> bzip2, xz, zstandard; schema-less Confluent Wire Format payloads.</li>
<li><strong>Feather V1:</strong> dictionary-encoded categorical arrays.</li>
<li><strong>Feather V2 / Arrow IPC:</strong> ZSTD buffers, Float16, nested Union/Map/Tensor buffers.</li>
</ul>
<p>(Feather V2 also has an internal Snappy-based fallback container used for the viewer's own transcoding. It is not a standard Arrow compression codec and is labelled that way.)</p>
<p>The tool page documents a 250 MB input safeguard and a 5,000,000-row ceiling. These are safeguards, not performance promises; codec, decoded size and browser memory all matter.</p>
<h2>Try an in-browser Parquet, Avro &amp; Feather viewer</h2>
<p>The engine powers the free <a href="https://kivarotools.com/developer/parquet-viewer">KivaroTools Parquet, Avro &amp; Feather Viewer</a>, which adds a column profiler, a Binary X-Ray inspector and three one-click sample datasets. Selected file contents are processed locally in your browser and are not uploaded to KivaroTools servers; the website may still load general analytics or other third-party resources.</p>
<p>If you have written a reader for any of these formats, which stage cost you the most time?</p>
]]></content:encoded></item><item><title><![CDATA[Engineering Browser-Based Utilities: A Deep Dive into KivaroTools]]></title><description><![CDATA[The engineering behind binary processing, memory management, application security, parsing, and financial calculations in a modern web utility platform.]]></description><link>https://kivarotools.hashnode.dev/engineering-browser-based-utilities-a-deep-dive-into-kivarotools</link><guid isPermaLink="true">https://kivarotools.hashnode.dev/engineering-browser-based-utilities-a-deep-dive-into-kivarotools</guid><dc:creator><![CDATA[Muhammad Ahmad]]></dc:creator><pubDate>Sat, 31 Oct 2026 01:17:32 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ae39971306486e1bf4be068/15a47c79-fd78-4674-bab9-1ff1d2672bf6.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Building a collection of online utilities is not simply a matter of putting forms and buttons on a webpage. Reliable tools require careful engineering across data processing, memory management, application security, numerical computation, and user experience.</p>
<p>While developing <strong>KivaroTools</strong>, I encountered engineering challenges that are easy to overlook when building small web applications. Processing large files in a browser, handling binary data correctly, protecting spreadsheet exports, managing potentially expensive regular expressions, and calculating accurate financial results all require deliberate design decisions.</p>
<p>This article explores some of those decisions, the implementation details identified during a source-code audit, and the limitations that remain important to acknowledge.</p>
<p>The goal is not to claim that every problem has been solved perfectly. It is to explain the engineering behind the platform and share lessons that may help other developers building browser-based applications.</p>
<p><strong>Explore the platform:</strong> <a href="https://kivarotools.com/">https://kivarotools.com/</a></p>
<hr />
<h2>1. Architecture: Separating Interfaces from Processing Logic</h2>
<p>KivaroTools uses a modern web application architecture built around TypeScript, React, Next.js App Router, and Tailwind CSS. Its production deployment uses Cloudflare Workers through the project's Vinext workflow.</p>
<p>A useful way to understand the architecture is to divide it into four conceptual layers:</p>
<ol>
<li><strong>Application and routing:</strong> Handles navigation, page structure, metadata, and the overall application experience.</li>
<li><strong>Tool interfaces:</strong> Provide input controls, validation messages, configuration options, and results.</li>
<li><strong>Processing engines:</strong> Contain reusable logic for operations such as encoding, parsing, file processing, and calculations.</li>
<li><strong>External integrations:</strong> Connect to services or data sources when a feature requires information beyond local processing.</li>
</ol>
<p>Separating processing logic from interface components makes individual tools easier to maintain and test. It also reduces the need to duplicate core algorithms across multiple pages.</p>
<p>For example, a Base64 utility should not need to implement its own binary conversion logic inside a React component. A dedicated processing module can handle the operation while the interface focuses on user interaction.</p>
<p>This separation is especially useful as a platform grows: shared engines can be improved independently of the pages that use them.</p>
<h2>2. Browser-Side Processing and Privacy Boundaries</h2>
<p>One of the important architectural decisions in KivaroTools is performing supported file and data operations in the browser.</p>
<p>When a tool processes a selected file locally, its contents do not need to be uploaded to an application server simply to perform that operation. Browser APIs provide several capabilities that make this possible:</p>
<ul>
<li><code>File</code> and <code>Blob</code> for handling user-selected files and binary data.</li>
<li><code>ArrayBuffer</code> for accessing binary content.</li>
<li><code>Uint8Array</code> and other typed arrays for working with bytes.</li>
<li><code>TextEncoder</code> and <code>TextDecoder</code> for text encoding and decoding.</li>
<li>Web Workers for moving selected computation away from the main interface thread.</li>
<li>Object URLs for making locally generated file content available to browser features.</li>
</ul>
<p>The source-code audit identified 32 core developer and document tools as using client-side processing for their main operations. This is a meaningful privacy benefit, but it should not be confused with a claim that the entire website makes no network requests.</p>
<p>Some features and supporting infrastructure can still use external services or network resources. The audit identified examples including:</p>
<ul>
<li>External foreign-exchange data.</li>
<li>Historical precious-metals data.</li>
<li>A metadata API.</li>
<li>PDF.js resources loaded from a CDN.</li>
<li>Optional analytics or tag-management integrations.</li>
<li>A Have I Been Pwned range API integration for password-related checks.</li>
</ul>
<p>The password-checking integration uses a partial SHA-1 hash prefix approach rather than sending the original password to the service.</p>
<p>The important distinction is between <strong>processing a selected file locally</strong> and <strong>the network activity of the entire website</strong>. They are not the same thing.</p>
<p>A privacy-conscious application should explain these boundaries clearly instead of making broad claims such as “nothing ever leaves your device.”</p>
<h2>3. Memory Management: Understanding Binary Data Costs</h2>
<p>Browser memory management becomes particularly important when working with large files.</p>
<p>Consider a typical file-processing workflow:</p>
<pre><code class="language-ts">const buffer = await file.arrayBuffer();
const bytes = new Uint8Array(buffer);
</code></pre>
<p>The first operation reads the file into an <code>ArrayBuffer</code>. The second creates a typed-array view over that buffer.</p>
<p>An important detail is that:</p>
<pre><code class="language-ts">const bytes = new Uint8Array(buffer);
</code></pre>
<p>does not automatically duplicate the entire underlying buffer. It creates a view over the existing memory.</p>
<p>However, additional processing steps may allocate more memory. For example:</p>
<ul>
<li>Creating transformed output buffers.</li>
<li>Converting binary data into Base64.</li>
<li>Building intermediate strings.</li>
<li>Combining multiple PDF files.</li>
<li>Retaining both input and output data during a transformation.</li>
</ul>
<p>This means peak memory usage can be significantly higher than the original file size, depending on the algorithm and runtime.</p>
<p>It is tempting to assign a fixed memory multiplier to every operation, but that would be misleading. JavaScript string representations, browser implementations, and intermediate allocations vary.</p>
<p>The source audit also identified that some file workflows read the complete input into memory rather than using a fully streaming pipeline. This makes memory consumption an important consideration for larger files.</p>
<p><strong>Engineering lesson:</strong> Optimize the complete data lifecycle, not just the final output size. Avoid unnecessary copies, release references when they are no longer needed, and consider streaming or chunked processing when the underlying format and implementation support it.</p>
<h2>4. Base64 Encoding: Why Chunk Boundaries Matter</h2>
<p>Base64 encoding converts binary data into a text representation. It is useful for data transport and embedding binary content, but large inputs require careful handling.</p>
<p>A common mistake is independently encoding arbitrary chunks and concatenating the resulting strings.</p>
<p>Base64 represents every three input bytes using four output characters. Therefore, independently encoded chunks should normally end on three-byte boundaries if their encoded strings are going to be concatenated directly.</p>
<p>Consider this chunk size:</p>
<pre><code class="language-ts">const chunkSize = 0x8000; // 32768 bytes
</code></pre>
<p>The problem is:</p>
<pre><code class="language-ts">32768 % 3 !== 0
</code></pre>
<p>A chunk of 32,768 bytes leaves a remainder of two bytes. Its independently generated Base64 representation can contain padding. Concatenating that padded output with the next independently encoded chunk does not produce the same result as encoding the original continuous byte sequence.</p>
<p>A chunk size divisible by three avoids that particular boundary problem:</p>
<pre><code class="language-ts">const chunkSize = 6144;

console.log(chunkSize % 3); // 0
</code></pre>
<p>The KivaroTools Base64 engine uses a 6,144-byte chunk size for its chunked conversion approach.</p>
<p>A simplified illustration of the boundary principle:</p>
<pre><code class="language-ts">function encodeChunks(bytes: Uint8Array): string {
  const chunkSize = 6144;
  const parts: string[] = [];

  for (let offset = 0; offset &lt; bytes.length; offset += chunkSize) {
    const chunk = bytes.subarray(
      offset,
      Math.min(offset + chunkSize, bytes.length)
    );

    let binary = "";

    for (let i = 0; i &lt; chunk.length; i++) {
      binary += String.fromCharCode(chunk[i]);
    }

    parts.push(btoa(binary));
  }

  return parts.join("");
}
</code></pre>
<p>This example illustrates the alignment principle. A production implementation must also account for browser runtime limits, conversion efficiency, error handling, and the treatment of the final chunk.</p>
<p>Text input introduces another important distinction. <code>btoa()</code> expects a binary string whose characters represent byte values; it is not a general-purpose UTF-8 text encoder.</p>
<p>For Unicode text, convert the string to bytes first:</p>
<pre><code class="language-ts">const bytes = new TextEncoder().encode("Hello, 世界");
</code></pre>
<p><strong>Engineering lesson:</strong> Binary data and text data are different representations. Correct encoding requires respecting that distinction.</p>
<h2>5. Object URLs and Resource Cleanup</h2>
<p>Browser-based file tools often use object URLs to make generated data available for previewing or downloading.</p>
<p>For example:</p>
<pre><code class="language-ts">const objectUrl = URL.createObjectURL(blob);
</code></pre>
<p>This creates a browser-managed reference to the underlying object. When the application no longer needs it, the URL should be released:</p>
<pre><code class="language-ts">URL.revokeObjectURL(objectUrl);
</code></pre>
<p>Failing to clean up object URLs can retain resources longer than necessary.</p>
<p>However, cleanup timing matters. Revoking a URL immediately after initiating a download or opening a preview may interfere with browser behavior in some workflows.</p>
<p>A robust implementation should manage the lifecycle according to the actual user interaction rather than assuming one fixed cleanup delay works in every browser.</p>
<p>This is another example of why browser APIs need to be treated as resource-management tools, not merely convenient functions.</p>
<h2>6. JSON Processing: Strict Validation and Controlled Repair</h2>
<p>JSON tools often need to support two different tasks:</p>
<ol>
<li>Determining whether input is valid JSON.</li>
<li>Helping users recover from common formatting mistakes.</li>
</ol>
<p>These tasks should not be confused.</p>
<p>Strict JSON validation should rely on the native parser:</p>
<pre><code class="language-ts">function isValidJson(input: string): boolean {
  try {
    JSON.parse(input);
    return true;
  } catch {
    return false;
  }
}
</code></pre>
<p>A repair feature is different. It may attempt to recover from selected common mistakes, but it should not silently treat arbitrary text as valid JSON.</p>
<p>The KivaroTools JSON formatter engine separates strict parsing from its optional repair process. The repaired output is parsed again with <code>JSON.parse()</code> before being accepted.</p>
<p>This final validation step is important: a successful repair transformation does not automatically mean the resulting text is valid JSON.</p>
<p>A reliable repair pipeline should also understand strings and escape sequences. For example, a comma inside a quoted string must not be treated as a structural separator.</p>
<pre><code class="language-json">{
  "message": "A comma, inside this string is valid",
  "active": true
}
</code></pre>
<p>A simplistic character replacement approach can easily corrupt otherwise valid content.</p>
<p>The broader engineering principle is to distinguish syntax recovery from semantic interpretation. A tool may be able to repair formatting, but it cannot always know what the user intended.</p>
<h2>7. Spreadsheet Formula Injection: A Security Detail Worth Taking Seriously</h2>
<p>Exporting data to CSV or spreadsheet formats introduces a security issue commonly known as <strong>CSV or Formula Injection</strong>.</p>
<p>A spreadsheet application may interpret cells beginning with certain characters as formulas rather than ordinary text.</p>
<p>Potentially dangerous prefixes include:</p>
<ul>
<li><code>=</code></li>
<li><code>+</code></li>
<li><code>-</code></li>
<li><code>@</code></li>
</ul>
<p>Control characters and full-width equivalents can also require consideration.</p>
<p>For example, a cell containing a formula-like value may be interpreted differently when opened in spreadsheet software.</p>
<p>The KivaroTools JSON converter includes a <code>safeSpreadsheetValue</code> helper that prefixes values beginning with several formula-related characters with an apostrophe. The audited implementation also considers tab, carriage return, line feed, and full-width variants.</p>
<p>A simplified illustration:</p>
<pre><code class="language-ts">function safeSpreadsheetValue(value: string): string {
  if (/^[=+\-@\t\r\n＝＋－＠]/.test(value)) {
    return `'${value}`;
  }

  return value;
}
</code></pre>
<p>This example demonstrates the basic idea, not a complete universal spreadsheet-security solution.</p>
<p>The audit identified an edge case: the check does not first trim leading whitespace. A value such as:</p>
<pre><code class="language-text"> =cmd
</code></pre>
<p>may therefore bypass an anchored prefix check.</p>
<p>The audit also identified a separate PDF-to-Excel helper that trims values before applying its check.</p>
<p>This highlights an important distinction: CSV and XLSX are different file formats, and their export paths may need different handling. An apostrophe prefix is not a universal guarantee across every spreadsheet application and import workflow.</p>
<p>Security checks should be designed and tested against the actual output format and target applications.</p>
<p><strong>Engineering lesson:</strong> Exported data is still untrusted input. Treat spreadsheet generation as a security-sensitive operation.</p>
<h2>8. Regular Expression Testing and ReDoS Mitigation</h2>
<p>Regular expressions are powerful, but certain patterns can cause excessive computation when evaluated against carefully chosen input.</p>
<p>This class of problem is known as Regular Expression Denial of Service, or ReDoS.</p>
<p>A vulnerable expression may take an unexpectedly long time to complete because of catastrophic backtracking.</p>
<p>One mitigation is to run potentially expensive regular-expression operations in a Web Worker. This keeps the main interface thread more responsive and allows the application to terminate the worker if it exceeds a time limit.</p>
<p>The audited KivaroTools regex tester uses an inline Blob-based Web Worker and a 1,200-millisecond watchdog.</p>
<p>Conceptually, the workflow is:</p>
<ol>
<li>Create a worker.</li>
<li>Execute the regular-expression operation away from the main UI thread.</li>
<li>Start a timeout.</li>
<li>Receive the result if execution finishes.</li>
<li>Terminate the worker if the timeout expires.</li>
</ol>
<p>However, a timeout is not a complete security guarantee. The worker can still consume CPU resources during the allowed execution period.</p>
<p>A robust regex-testing tool should also consider:</p>
<ul>
<li>Maximum pattern length.</li>
<li>Maximum input length.</li>
<li>Result and match-count limits.</li>
<li>Worker cleanup.</li>
<li>Clear timeout feedback.</li>
<li>Avoiding unsafe synchronous fallback execution.</li>
</ul>
<p><strong>Engineering lesson:</strong> Moving computation into a worker improves responsiveness and provides a termination boundary, but it does not make arbitrary regular expressions harmless.</p>
<h2>9. SQL Formatting: Formatting Is Not Semantic Validation</h2>
<p>SQL formatting helps developers read and maintain queries by applying consistent indentation, capitalization, and layout.</p>
<p>KivaroTools uses <code>sql-formatter</code> version 15.9.0 and the audited configuration supports 19 SQL dialects.</p>
<p>Formatting and validation are different operations.</p>
<p>A formatter can improve the presentation of a query without proving that:</p>
<ul>
<li>The query is syntactically valid for every database.</li>
<li>Referenced tables and columns exist.</li>
<li>The query has the intended business meaning.</li>
<li>The database will execute it successfully.</li>
</ul>
<p>The platform also includes heuristic linting checks, but these should not be described as a complete SQL compiler or semantic validator.</p>
<p>This distinction matters because SQL dialects differ. A query that is accepted by one database may not behave the same way in another.</p>
<p>A useful developer tool should communicate what it actually checks rather than implying that formatted SQL is automatically correct SQL.</p>
<h2>10. Salary Calculators: Accuracy Requires Jurisdiction-Specific Rules</h2>
<p>Financial calculators introduce a different class of engineering challenge.</p>
<p>Unlike a text formatter, a salary calculator must account for rules that vary by jurisdiction, tax year, income type, deductions, contributions, and regional circumstances.</p>
<p>The KivaroTools salary calculator audit distinguishes between two implementation categories:</p>
<ul>
<li><strong>16 jurisdictions</strong> with independently verified statutory models in the audit.</li>
<li><strong>6 jurisdictions</strong> using disclosed benchmark approximations for certain regional taxes or social contributions.</li>
</ul>
<p>The six approximation jurisdictions identified in the audit are:</p>
<ul>
<li>Canada</li>
<li>France</li>
<li>Japan</li>
<li>Switzerland</li>
<li>Italy</li>
<li>Mexico</li>
</ul>
<p>Regional rules can make a substantial difference. For example, calculations may depend on provincial circumstances in Canada, prefectural or municipal rules in Japan, cantonal differences in Switzerland, or regional and municipal components in Italy.</p>
<p>A benchmark approximation can be useful as an estimate, but it should not be presented as an exact calculation for every individual taxpayer.</p>
<p>Tax rules also change. A model that was accurate for one tax year may become outdated when legislation, contribution thresholds, or statutory rates change.</p>
<p>For that reason, transparent disclosures and regular verification are essential parts of financial software engineering.</p>
<p><strong>Engineering lesson:</strong> A financial calculator is only as trustworthy as its assumptions, source data, and tax-year coverage.</p>
<h2>11. Reverse Salary Calculations and Numerical Solvers</h2>
<p>Some salary tools solve the reverse problem: instead of calculating take-home pay from gross salary, they estimate the gross salary required to achieve a target net amount.</p>
<p>This is a numerical search problem.</p>
<p>A common approach is binary search or bisection. Given a valid interval and a suitably monotonic function, the algorithm repeatedly narrows the range containing the target.</p>
<p>A simplified illustration:</p>
<pre><code class="language-ts">function findGrossSalary(
  targetNet: number,
  low: number,
  high: number,
  calculateNet: (gross: number) =&gt; number
): number {
  let best = low;

  for (let i = 0; i &lt; 60; i++) {
    const mid = (low + high) / 2;
    const net = calculateNet(mid);

    best = mid;

    if (Math.abs(net - targetNet) &lt; 0.005) {
      return mid;
    }

    if (net &lt; targetNet) {
      low = mid;
    } else {
      high = mid;
    }
  }

  return best;
}
</code></pre>
<p>This is an illustrative example, not a replacement for the platform's production implementation.</p>
<p>The audit identified two reverse-calculation paths:</p>
<ul>
<li>A salary calculator path with up to 50 iterations and a tolerance of 0.5.</li>
<li>An advanced salary-analysis path with up to 60 binary-search iterations and an early threshold of 0.005.</li>
</ul>
<p>These iteration counts alone do not guarantee arbitrary sub-cent accuracy.</p>
<p>The result also depends on:</p>
<ul>
<li>Whether the search interval brackets a solution.</li>
<li>Whether the net-pay function is monotonic over the interval.</li>
<li>Floating-point precision.</li>
<li>Rounding rules.</li>
<li>Tax brackets and contribution thresholds.</li>
<li>Discontinuities or benefit cliffs.</li>
</ul>
<p>For example, if a calculation includes a threshold-based deduction or benefit, the function may not behave like a perfectly smooth curve.</p>
<p><strong>Engineering lesson:</strong> Numerical algorithms need valid assumptions, appropriate bounds, and independent tests. An iteration count is not an accuracy guarantee.</p>
<h2>12. Automated Testing: What the Audit Reported</h2>
<p>Automated testing is essential for a platform with many independent utilities and calculation engines.</p>
<p>The source-code verification work reported the following results for selected test suites:</p>
<table>
<thead>
<tr>
<th>Area</th>
<th align="right">Reported result</th>
</tr>
</thead>
<tbody><tr>
<td>Tool registry</td>
<td align="right">63 routes</td>
</tr>
<tr>
<td>Base64</td>
<td align="right">63/63</td>
</tr>
<tr>
<td>JSON converter</td>
<td align="right">199/199</td>
</tr>
<tr>
<td>Regex tester</td>
<td align="right">248/248</td>
</tr>
<tr>
<td>SQL formatter</td>
<td align="right">42/42</td>
</tr>
<tr>
<td>PDF tools</td>
<td align="right">211/211</td>
</tr>
<tr>
<td>Mortgage calculator</td>
<td align="right">73/73</td>
</tr>
<tr>
<td>EMI calculator</td>
<td align="right">209/209</td>
</tr>
<tr>
<td>Compound interest</td>
<td align="right">234/234</td>
</tr>
<tr>
<td>SIP calculator</td>
<td align="right">285/285</td>
</tr>
<tr>
<td>SVG optimizer</td>
<td align="right">286/286</td>
</tr>
</tbody></table>
<p>The audit also reported:</p>
<ul>
<li>12 test suites passing cleanly.</li>
<li>ESLint reporting zero errors and 21 warnings.</li>
<li>The master salary verifier reporting 3,152 passing assertions out of 3,156.</li>
</ul>
<p>The four remaining salary assertions were associated in the audit with possible fixture drift involving Pakistan Finance Act 2024 tax slabs. That explanation should be treated as a hypothesis until the expected values are independently checked against the applicable statutory rules.</p>
<p>A test suite passing is useful evidence, but it does not establish that every possible input or real-world situation is correct.</p>
<p>Testing should include edge cases, regression tests, independent reference calculations, and updated statutory fixtures.</p>
<h2>13. Production Deployment and Route Verification</h2>
<p>KivaroTools is deployed on Cloudflare Workers using the project's Vinext workflow.</p>
<p>The verification report also described production probes against selected routes, including:</p>
<ul>
<li>The homepage.</li>
<li>PDF merge and split pages.</li>
<li>JSON formatting.</li>
<li>Salary calculation pages.</li>
</ul>
<p>The selected route checks reportedly returned successful HTTP responses and included structured metadata checks.</p>
<p>These checks are useful for confirming that selected production routes respond as expected.</p>
<p>However, a successful HTTP response does not prove that every interactive feature works correctly. Browser-side behavior, file processing, downloads, and user interaction still require dedicated testing.</p>
<p>This is why production route verification and manual staging QA should be treated as complementary activities rather than interchangeable ones.</p>
<h2>14. Engineering Lessons and Areas for Improvement</h2>
<p>Building browser-based utilities has reinforced several practical lessons.</p>
<p><strong>1. Privacy needs precise boundaries.</strong><br />Local file processing is a meaningful benefit, but external data services, analytics, and supporting resources should be disclosed accurately.</p>
<p><strong>2. Memory is part of the user experience.</strong><br />Large files can create significant memory pressure. Avoid unnecessary allocations and consider streaming where practical.</p>
<p><strong>3. Binary encoding requires mathematical correctness.</strong><br />Chunk boundaries, padding, and text encoding are not implementation details to ignore. Small mistakes can corrupt large outputs.</p>
<p><strong>4. Security checks need adversarial test cases.</strong><br />Spreadsheet exports and regular-expression tools should be tested against hostile or unusual inputs, not just normal examples.</p>
<p><strong>5. Formatting is not validation.</strong><br />A well-formatted SQL query or repaired JSON document is not automatically semantically correct.</p>
<p><strong>6. Financial calculations require transparent assumptions.</strong><br />Statutory models, regional approximations, tax years, and rounding behavior should be clearly distinguished.</p>
<p><strong>7. Test results need context.</strong><br />Passing assertions are valuable, but unresolved failures should remain visible until their expected values have been independently verified.</p>
<p>These lessons are not unique to KivaroTools. They apply to many applications that process user data directly in the browser.</p>
<h2>15. What Comes Next</h2>
<p>The engineering work does not end when a tool becomes available online.</p>
<p>Several areas deserve continued attention:</p>
<ul>
<li>Improving memory efficiency for large-file workflows.</li>
<li>Evaluating streaming approaches where supported by the relevant formats.</li>
<li>Strengthening spreadsheet-export edge-case handling.</li>
<li>Continuing to test regular-expression execution limits.</li>
<li>Reviewing numerical solvers against independent reference cases.</li>
<li>Updating jurisdiction-specific financial assumptions as legislation changes.</li>
<li>Expanding browser and device compatibility testing.</li>
<li>Improving documentation around data processing and external integrations.</li>
</ul>
<p>The objective is not simply to increase the number of tools. It is to make the existing tools more reliable, understandable, and useful.</p>
<h2>Conclusion</h2>
<p>Building KivaroTools has been a practical exercise in applying software engineering principles to everyday web utilities.</p>
<p>Behind a simple interface can be complex decisions about binary data, memory allocation, parsing, application security, numerical methods, and privacy boundaries.</p>
<p>The most valuable lesson is that reliability comes from understanding those details and being honest about their limitations.</p>
<p>A browser-based utility should not only produce an output. It should process data responsibly, communicate its assumptions, handle errors predictably, and make its behavior understandable to the person using it.</p>
<p>That is the engineering direction behind KivaroTools.</p>
<p><strong>Explore KivaroTools:</strong> <a href="https://kivarotools.com/">https://kivarotools.com/</a></p>
<hr />
<p><em>Technical note: This article reflects findings reported in the KivaroTools source-code verification audit dated October 31, 2026. Test results and implementation details are audit-reported results, not a guarantee that every future release or every possible input has been independently verified. Financial calculations may use statutory models or disclosed benchmark approximations depending on jurisdiction. Readers should review the relevant tool disclosures and independently verify results before relying on them for production or financial decisions.</em></p>
]]></content:encoded></item></channel></rss>