<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[amailuk]]></title><description><![CDATA[Learn practical data cleaning, analysis, and automation with Python, SQL, PostgreSQL, Excel, and Power BI. Explore tutorials, reporting tips, and real-world data solutions.]]></description><link>https://amailuk.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 01 Nov 2026 01:17:53 GMT</lastBuildDate><atom:link href="https://amailuk.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Writing Is a Way of Life]]></title><description><![CDATA[A personal reflection on how writing, data analysis and technology connect through curiosity, research, storytelling and everyday problem-solving.]]></description><link>https://amailuk.hashnode.dev/writing-is-a-way-of-life</link><guid isPermaLink="true">https://amailuk.hashnode.dev/writing-is-a-way-of-life</guid><category><![CDATA[writing services]]></category><dc:creator><![CDATA[amailuk]]></dc:creator><pubDate>Thu, 29 Oct 2026 12:16:34 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ace4dffa8f3b5e2c75d0763/67e05219-ade9-4e8e-8fae-4417a9bc0769.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h1>Writing Is a Way of Life</h1>
<p>Long before I started thinking about careers, making a living or figuring out what I wanted to do with my life, I was already writing. As children, my siblings, cousins, friends and I would hold writing competitions, and I enjoyed every bit of it. I would later find myself writing letters for friends, <em>becoming something of a ghostwriter</em> before I even knew it had a name to it. Looking back, I realise that writing was there long before I began thinking seriously about what it could mean for my future.</p>
<p>I did not have a grand plan. I simply loved writing. Over time, that love found its way into teaching, research, data analysis and, in my interest in technology. These may seem like different pursuits, but I keep finding myself returning to the same thing: the desire to explore ideas, make sense of them and communicate what I discover.</p>
<p>Perhaps that is why I struggle to think of writing as just another skill. It has become part of how I experience life, and I am increasingly interested in building a life around it.</p>
<h2>My Early Love for Words</h2>
<p>Eventually, I ventured into teaching, where I took up Literature. It was a natural fit. Being around books, reading stories and exploring ideas only deepened my love for writing. Over the years, writing has remained my way of expressing myself, communicating with others and making sense of the world around me.</p>
<p>As someone who enjoys solitude, it has also made life as a recluse a little more interesting. I can spend hours with my thoughts, putting them into words, and never really feel alone.</p>
<p>Growing up, I also found myself writing letters for my friends. In a way, <em>I was their ghostwriter</em>, helping them express what they wanted to say while enjoying the opportunity to put my own words to work. Every opportunity to write felt like a blessing. I got to make fellow students happy, and in the process, I discovered that helping someone else find the right words could be just as rewarding as writing for myself.</p>
<p>Looking back, perhaps this was one of the earliest signs that writing would remain a constant in my life. The setting have changed over the years, but the enjoyment I get from finding the right words and helping others communicate has stayed with me.</p>
<h2>When Writing Meets Data Analysis</h2>
<p>My interests have since extended into research and data analysis, but writing remains the thread that runs through them all.</p>
<p>Interestingly, I can trace part of my interest in data analysis back to my love of writing. After all, analysing data is not always the end of the job. You often have to make sense of what you have found and communicate it through a clear, formal report. The findings might be interesting, but they are of limited use if you cannot explain what they mean and why they matter.</p>
<p>I have also helped a number of people document their findings, especially those who would much rather do the analysis than write about it. I understand that writing does not come naturally to everyone, but I enjoy helping people turn their findings into something others can read, understand and use.</p>
<p>In some ways, this takes me back to those days when I wrote letters for my friends. The setting though different, the same underlying idea is familiar: helping people communicate what they might otherwise struggle to express.</p>
<h2>Data Has Stories to Tell, Too</h2>
<p>Data itself is fun. It tells stories, much like writing does. It gives you something to investigate, questions to ask and patterns to discover. There are mysteries hidden in the numbers, and part of the enjoyment comes from working through them until something begins to make sense.</p>
<p>Of course, data analysis has its limits. What you can conclude depends on the information available and what it can reasonably tell you. Still, there is plenty of room for curiosity, discovery and the satisfaction of finding something you had not noticed before.</p>
<p>For me, writing and data analysis are natural partners. One allows me to explore and communicate ideas through words, while the other gives me a way to investigate questions and uncover insights. Data gives me something to investigate and report on, while writing helps me communicate what I have discovered.</p>
<p>Together, they have given me another way to learn, express myself and make my working life more enjoyable.</p>
<h2>Technology: Another World to Explore</h2>
<p>Then there is technology. Honestly, I am still figuring out exactly where it fits into all of this. I find it difficult to separate it neatly from my other interests because it seems to weave its way into almost everything I do.</p>
<p>Sometimes, it feels like the glue holding the different pieces together. At other times, it pulls me into a world of its own, giving me more questions to ask and more things to explore. And being naturally curious, I find that hard to resist.</p>
<p>What I appreciate most about technology is the way it opens up possibilities. Things that would otherwise take ages, require considerable effort or simply be out of reach suddenly become possible. </p>
<p>It makes me wonder about all the things we can do today that would have been difficult to imagine a few years ago. I also find myself wondering how I managed without some of the tools I now use. It is funny how quickly something that once seemed unfamiliar becomes part of your everyday life.</p>
<p>I particularly enjoy how technology fits into my love for writing and my interest in data analysis. It gives me more ways to research, work with information, organise my thoughts and communicate what I discover. It also introduces me to possibilities beyond writing and analysing data.</p>
<p>I can explore ways of building tools that make a task easier or help someone solve a problem. These do not have to be elaborate creations. Sometimes, a simple tool that saves someone time or makes their work less frustrating is useful enough. I like that possibility: not just learning about technology for its own sake, but finding ways to put it to work.</p>
<p>I am still exploring what I can do with it, and I do not pretend to have everything figured out. That is part of the appeal. There is always something new to learn, something to question and another way to approach a problem. I can follow an idea simply because I want to understand how something works, and sometimes that curiosity leads me somewhere I had not planned to go.</p>
<p>For someone who enjoys research and writing, having another world full of things to investigate is hardly a bad thing.</p>
<h2>Writing About Technology, Too</h2>
<p>I also want to do more writing about technology: how it affects our lives, why it matters and how we can make it work better for us.</p>
<p>I am interested not only in the tools themselves, but also in what they mean for the way we work, learn and live. Technology can feel complicated, but I believe there is value in explaining things in ways people can relate to and use. Writing gives me a familiar way to explore these questions and share what I learn.</p>
<p>Perhaps that is where technology connects most naturally with the rest of my life. It gives me more to investigate, more to write about and more opportunities to turn ideas into something useful.</p>
<p>I do not need to claim that I am an expert in everything technology touches. I am simply curious, interested in learning and keen to see what I can do with the knowledge I gain. For now, that is enough to keep me exploring.</p>
<h2>Why I Choose the Writer's Life?</h2>
<p><em>Why pursue the writer's life</em>? Well, it fits into who I am and how I want to live.</p>
<p>There is something appealing about being able to earn a living doing something I genuinely enjoy. Writing rarely feels like a chore once I get into it. More often than not, I find myself wanting to continue long after I should have stopped. Many times, word limits are the only things that force me to call it a day, even when I still have more to say.</p>
<p>I have always dreamed of becoming a writer. At one point, I tried my hand at novel writing, but it did not quite work out as I had hoped. That has not put me off writing stories, though. I still intend to explore short-story writing, which feels like a natural next step for me.</p>
<p>There is always something to learn, another idea to pursue and another story waiting to be told.</p>
<h2>The Freedom to Work on My Own Terms</h2>
<p>I am not particularly attached to a desk or a conventional office setting. I can manage one when I need to, but I prefer open spaces and a bit of freedom.</p>
<p>Give me my laptop, a reliable internet connection and the opportunity to write, and I am comfortable. I could live in almost any place where I have those basics and still do what I love. The same flexibility appeals to me in research and data analysis: the opportunity to explore, investigate and produce something useful without necessarily being tied to one particular setting.</p>
<p>Technology makes that kind of flexibility even more practical. It gives me access to tools, information and opportunities that make it easier to work from different places and pursue different interests.</p>
<p>For me, this is part of the attraction. I want a working life that gives me room to think, explore and create, rather than one that confines me to a particular setting or routine.</p>
<h2>Building a Life Around What I Love</h2>
<p>Writing comes naturally to me, so it makes sense to give that ability a proper place in my life. I want to use it not only to sustain myself financially, but also to create the freedom to live on my own terms.</p>
<p>Beyond that, I hope my words can be useful to others in some way, whether by informing, encouraging, entertaining or helping someone see things differently.</p>
<p>My interest in data analysis adds another dimension to that ambition: the opportunity to turn information into insights that people can understand and act upon. Technology, meanwhile, gives me more ways to develop these interests and put them to practical use.</p>
<p>Ultimately, I want to build a life around things I genuinely enjoy rather than treat them as activities I pursue only when everything else is finished. I want to keep learning, researching, analysing data, exploring technology, telling stories and finding new ways to put thoughts into words.</p>
<p>I do not need to make any grand claims about where this will take me. I know only that these interests give me something meaningful to pursue, a way to earn a living and the possibility of making a difference in the lives of others.</p>
<h2>Writing Is Me</h2>
<p>I suppose what I am trying to say is that writing is part of who I am. It feels so deeply ingrained in me that I struggle to imagine life without it. I might even say that writing is part of my DNA, though I know that sounds a little clichéd. Still, it captures how I feel.</p>
<p>Writing is not simply something I do; it is one of the ways I experience and make sense of life. Data analysis has become another way of doing much the same thing, through questions, patterns and discoveries. Technology helps me bring these interests together, while also giving me new things to learn and explore.</p>
<p>For me, the three belong together. They make my work more interesting, give my curiosity somewhere to go and make life a little more fun. Writing helps me express ideas, data analysis helps me investigate them, and technology gives me more ways to explore, create and share what I discover.</p>
<p>Perhaps you should try bringing them together sometime. You might be surprised by what you discover, not only about the world around you, but also about yourself.</p>
<h2>Have a Writing Task? Send It My Way.</h2>
<p>Unstuck yourself from the writing task sitting on your desk—or somewhere in your to-do list—feel free to send it my way. It could be a technical article, a guide that needs to make sense to everyday readers, documentation that needs a little clarity, or data findings that need to be turned into a report people can actually understand.</p>
<p>I enjoy the process of taking ideas, making sense of them and finding the right words to bring them to life. Technology writing is an area I am keen on, and I welcome opportunities to put my writing skills to use. Learning along the way and facilitating others, to communicate their ideas better.</p>
<p>So, the next time you have a writing task that needs doing, think of me. You bring the subject; I will bring my curiosity, a willingness to dig deeper and my love for finding the right words. </p>
<p>Let's see what we can make of it together.</p>
]]></content:encoded></item><item><title><![CDATA[Data Cleaning Standards]]></title><description><![CDATA[Data cleaning makes data reliable, not just tidy. Blind changing of values distorts data. Pus, consistency does not automatically guarantee accuracy]]></description><link>https://amailuk.hashnode.dev/data-cleaning-standards</link><guid isPermaLink="true">https://amailuk.hashnode.dev/data-cleaning-standards</guid><category><![CDATA[data cleaning ]]></category><category><![CDATA[standards]]></category><category><![CDATA[Data Science]]></category><category><![CDATA[cleaning services]]></category><dc:creator><![CDATA[amailuk]]></dc:creator><pubDate>Tue, 20 Oct 2026 08:07:43 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ace4dffa8f3b5e2c75d0763/dfe2f383-b3be-4fa1-98ed-fae7f004b597.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2>1. Why standards matter</h2>
<p>Picture opening a spreadsheet you've just been handed. It's full. Hundreds of rows, all neat in their little cells. You feel good. The hard part's done, right?</p>
<p>Not so fast.</p>
<p>I've been there. You start counting, and the numbers feel off. You group sales by product and get three lines for what's clearly one item. You sort by date and the order makes no sense. You average a column and the answer looks way too low, because half the rows are empty.</p>
<p>The data was <em>there</em>. It just wasn't <em>ready</em>.</p>
<p>Here's what usually hides in a "full" dataset:</p>
<ul>
<li><p><strong>Dates in mixed formats.</strong> One row says <code>2026-10-15</code>. The next says <code>15/10/2026</code>. Same day. Two spellings.</p>
</li>
<li><p><strong>Product names that don't match.</strong> <code>Apple</code>, <code>apple</code>, and <code>Apple</code> (with a sneaky space at the end) look the same to you. To a computer, they're three different things.</p>
</li>
<li><p><strong>Missing values.</strong> Some break your code. Others sneak through and quietly bend your math.</p>
</li>
</ul>
<p>Think of it like a kitchen where every cook labels the spice jars their own way. The jars are all full. But good luck finding the cumin at 7 p.m. with guests at the door.</p>
<p>That's the point of this article. Data can be present without being ready for use. Standards are what close that gap.</p>
<hr />
<h2>2. What are data standards?</h2>
<p>A data standard is just an agreement. It's a rule everyone follows so the same thing always looks the same way.</p>
<p>That's it. No magic. Think of road rules. Everybody drives on the same side, so nobody has to guess. Data standards do that for your columns.</p>
<p>They usually cover four areas:</p>
<ul>
<li><p><strong>Format standards</strong> decide <em>how a value is represented.</em> Is a date written as <code>2026-10-30</code> or <code>30/10/2026</code> maybe <code>10/30/2026</code>? Pick one.</p>
</li>
<li><p><strong>Naming standards</strong> decide <em>how labels and categories are written.</em> Is it <code>Apple</code>, <code>apple</code>, or <code>APPLE</code>? Pick one.</p>
</li>
<li><p><strong>Validation standards</strong> decide <em>which values are acceptable.</em> Can a quantity be negative? Can an order date be in the year 2087? Spell it out.</p>
</li>
<li><p><strong>Missing-data standards</strong> decide <em>how absent or unknown values are shown and handled.</em> Is it a blank? <code>N/A</code>? <code>Unknown</code>? Pick one, and know what it means.</p>
</li>
</ul>
<p>There's a catch. <strong>Standards give you consistency. They don't give you accuracy.</strong></p>
<p>You can write every price in the same tidy format and still have the wrong prices. A neat wrong answer is still wrong. Keep that in your back pocket. We'll come back to it.</p>
<hr />
<h2>3. The problem of inconsistent data</h2>
<p>So what actually goes wrong when data is messy? Let's look at it.</p>
<table>
<thead>
<tr>
<th>Raw values</th>
<th>Potential problem</th>
</tr>
</thead>
<tbody><tr>
<td><code>Apple</code>, <code>apple</code>, <code>Apple</code></td>
<td>One product may appear under several labels.</td>
</tr>
<tr>
<td><code>2026-10-15</code>, <code>15/10/2026</code></td>
<td>Dates may be interpreted inconsistently.</td>
</tr>
<tr>
<td>Blank, <code>N/A</code>, <code>Unknown</code></td>
<td>Missing or unknown information may be treated differently.</td>
</tr>
<tr>
<td>Repeated order IDs</td>
<td>Records may be double-counted or the wrong record retained.</td>
</tr>
</tbody></table>
<p>Each row in that table is a small trap. Let's walk through them.</p>
<p><strong>The product that split in three.</strong> Say you want to know how many apples you sold this quarter. You group the sales by product name. But your data has Apple, apple, and Apple (that last one has a sneaky space at the end). To pandas (the computer), those are three different products. So your report shows three separate lines, each with a small number next to it. Let's say 40, 35, and 25 respectively.</p>
<p>Now picture your manager reading that report. She sees three tiny products. None of them looks like a winner. What she can't see is that they're really one product with 100 sales, maybe your best seller. So she says, "Apples aren't moving. Let's stop stocking them." And just like that, your top product gets cut. Not because customers stopped buying it. Because the data spelled it three different ways. That's a real business decision, made because of typos.</p>
<p><strong>The date that means two things.</strong> Is <code>03/04/2026</code> the 3rd of April or the 4th of March? Depends who typed it. If your tool guesses, it'll guess the same way for everything. Even when half the rows meant the opposite.</p>
<p><strong>The blank that isn't the same as "unknown."</strong> A blank might mean "we forgot." <code>N/A</code> might mean "doesn't apply." <code>Unknown</code> might mean "we tried and couldn't find out." Those are three different stories. If you treat them as one, you lose the story.</p>
<p>Three different ways of saying "nothing here." Imagine a customer list with a column for phone numbers. Three customers have no number on file, but each one is marked differently:</p>
<p>Sam's phone number is left blank. Nobody ever asked him. The person at the desk was busy and moved on.</p>
<p>Maria's says N/A. That's short for "not applicable." She told us she doesn't own a phone, so there's no number to collect.</p>
<p>Joe's says Unknown. We asked him, twice. He didn't want to share it, or we lost the note. Either way, we tried and came up empty.</p>
<p><strong>The order that shows up twice.</strong> Same order ID, two rows. Count both, and your revenue is inflated. Keep the wrong one, and you've got an old price or an old address in your report.</p>
<p>None of this crashes anything. That's what makes it scary. The report still gets built. The chart still looks great. And somebody still makes a decision based off it.</p>
<p>Inconsistency creates extra work. Worse still, it can mislead you.</p>
<hr />
<h2>4. Applying standards in practice with Pandas</h2>
<p>Alright, let's get our hands dirty. This is where Pandas, a popular Python library for working with tables, comes in.</p>
<p>A quick note before we start. I'm not going to just show you code. I want to show you the <em>decision</em> behind each line. Because the code is the easy part. Knowing <em>why</em> you're writing it is the skill.</p>
<h3>A. Standardizing text</h3>
<pre><code class="language-python">df["product_name"] = (
    df["product_name"].str.strip().str.lower()
)
</code></pre>
<p>Two things happen here. <code>.str.strip()</code> trims the invisible spaces at the start and end of each value. <code>.str.lower()</code> turns everything in the string to owercase. So <code>Apple</code> , <code>apple</code>, and <code>APPLE</code> all become <code>apple</code>.</p>
<p>Handy, right? But don't get too comfortable.</p>
<p>Lowercasing fixes <em>spelling style.</em> It doesn't fix <em>meaning.</em> If one row says <code>apple</code> and another says <code>apples</code>, or <code>green apple</code> and <code>granny smith</code>, this code does nothing. A human still has to decide whether those are the same product. Then you write a rule for it, maybe a lookup table that maps every variation to one official name.</p>
<p>So ask yourself first: <em>what should "the same product" mean here?</em> Then reach for the code.</p>
<h3>B. Standardizing dates</h3>
<pre><code class="language-python">df["order_date"] = pd.to_datetime(
    df["order_date"], errors="coerce", dayfirst=True
)
</code></pre>
<p>Why bother turning dates into real dates? Because text can't do date math. You can't properly sort <code>"15/10/2026"</code> as text. You can't count days between orders. Once it's a true datetime value, all of that just works fine.</p>
<p>Now look at <code>dayfirst=True</code>. That's a decision, not a detail. It tells Pandas, "when a date is unclear, read the day first." Is that right for your data? Only the data-clerk can say for a fact. If your data came from a country or system that writes month first, this setting will scramble things. Use if you are a day first region.</p>
<p>And look at <code>errors="coerce"</code>. It means "if you can't read a value, don't crash. Turn it into <code>NaT</code>" (that's "Not a Time," the date version of "missing" value). That's a kind gesture from Pandas. But it's also a warning flag. Every <code>NaT</code> was a value that <em>didn't make sense.</em> Check them out. Don't just let them slide.</p>
<h3>C. Handling missing values</h3>
<pre><code class="language-python">missing = df["order_date"].isna()
</code></pre>
<p>This one line gives you a True/False answer for every row. True means "this date is missing." Easy to write. The real question is what you do next.</p>
<p>And here's where I'll be honest: there's no single right answer.</p>
<ul>
<li><p>Is the missing field <em>needed</em> for your analysis? Maybe you can leave those rows out of this one calculation.</p>
</li>
<li><p>Can you <em>fill it in</em> from another trustworthy source?</p>
</li>
<li><p>Or is it a sign something broke upstream? Maybe the form let people skip the field. Maybe an import failed.</p>
</li>
</ul>
<p>Skipping a missing value feels like the tidy move. But sometimes it's the lazy one. If 40% of your dates are missing, that isn't a cleanup chore. That's a story you need to chase down.</p>
<p>Treat a missing value like a smoke alarm. Sometimes it's burnt toast. Sometimes it's a real fire. You don't know until you bother to look.</p>
<p>You could still determine the percentage of values missing</p>
<pre><code class="language-python">df = (df.isna().sum() / len(sales_data))*100
</code></pre>
<p>You sum the column's missing values, divide by the number of rows in the dataframe. Multiply by <code>100</code> to give you the percentage of the missing values. On average, anything over 40% in missing values may mean you cannot expect an accurate representation of the field.</p>
<h3>D. Managing duplicates</h3>
<pre><code class="language-python">df = df.sort_values("updated_at")
df = df.drop_duplicates(
    subset="order_id", keep="last"
)
</code></pre>
<p>Here's the logic. First we sort by <code>updated_at</code>, oldest to newest. Then we drop repeated <code>order_id</code> values, keeping the <em>last</em> one. In plain words: "If an order shows up more than once, keep the newest version."</p>
<p>Sounds smart. But it only works if two things are true:</p>
<ol>
<li><p>The <code>updated_at</code> column <em>reliably</em> tells you which record is newest. (Is it filled in every time? Is it accurate? Or could it be the date someone ran an export?)</p>
</li>
<li><p>Your business rules say the newest record <em>should</em> win. (Maybe in your company the first record is the official one, and later edits need approval.)</p>
</li>
</ol>
<p>If either one's shaky, this tidy little code deletes the wrong rows. And deleted rows don't send you an apology note.</p>
<p><strong>The big takeaway:</strong> Pandas gives you the tools. The standard tells you how to use them.</p>
<hr />
<h2>5. Standards require judgment, not blind automation</h2>
<p>It's tempting to write one cleaning recipe and run it on every column. One loop. Done by lunch.</p>
<p>Please don't. I've seen what happens.</p>
<p>Say you lowercase every text column. Product labels? Fine. But what about a tracking code like <code>aB3xK9</code>? In some systems, case matters. <code>aB3xK9</code> and <code>ab3xk9</code> could be two different packages. Lowercase them all, and you may have just merged two real records into one. Or broken your link to another system.</p>
<p>Or say you replace every missing value with zero. Looks clean and efficient, too! But a <em>missing price</em> and a <em>price of zero</em> aren't the same thing.</p>
<ul>
<li><p>One says "we don't know."</p>
</li>
<li><p>The other says "it was free."</p>
</li>
</ul>
<p>Fill in the zeros and your average price drops, and nobody can tell why.</p>
<p>It's like using one wrench for every bolt. Sometimes it fits. Sometimes you strip the thread.</p>
<p>So before you touch any column, slow down and ask three questions:</p>
<ol>
<li><p><strong>What does this field represent?</strong> A name? A code? A count? A date? A rating? What does a blank mean here?</p>
</li>
<li><p><strong>What rule fits that field?</strong> Not the rule that's easiest. The rule that matches what the field is.</p>
</li>
<li><p><strong>How will I check that the rule didn't hurt the data?</strong> Before you start, know what "still fine" looks like.</p>
</li>
</ol>
<p>That third question is the one people skip. We'll give it its own section next.</p>
<p>Good standardization retains the meaning and improves the consistency. If you lose the first to get the second, you haven't cleaned anything. You've just made it look nicer.</p>
<hr />
<h2>6. Checking whether the standards worked</h2>
<p>You ran your cleaning code. No errors. It looks good. So we're done, right?</p>
<p>Nope.</p>
<p>A transformation isn't proof of quality. It's just proof that the code ran.</p>
<p>Think about proofreading. You'd never write an essay, fix a few typos, and hit "send" without reading it again. Data deserves the same respect.</p>
<p>Here's a simple checklist I like to run after cleaning:</p>
<ul>
<li><p><strong>Remaining missing values.</strong> How many are left? Is that number expected? Did any <em>new</em> ones appear?</p>
</li>
<li><p><strong>Unexpected formats.</strong> Do all your dates look like dates? Do your IDs still follow the pattern they should?</p>
</li>
<li><p><strong>Duplicate identifiers.</strong> Is each <code>order_id</code> now truly unique? Count them.</p>
</li>
<li><p><strong>Invalid dates.</strong> How many <code>NaT</code> values did the conversion create? Why?</p>
</li>
<li><p><strong>Changes in record counts.</strong> How many rows did you start with? How many now? If say you lost 500, can you explain <em>exactly</em> why?</p>
</li>
</ul>
<p>Then do the step most people miss: <strong>compare against the original.</strong> Keep a copy of the raw data. Line it up next to the cleaned version. Look at what changed. Look at what <em>shouldn't</em> have changed but did.</p>
<p>A few small checks go a long way:</p>
<pre><code class="language-python">print(len(original), len(df))           # Did the row count change?
print(df["order_id"].is_unique)         # Are IDs now unique?
print(df["order_date"].isna().sum())    # How many dates are still missing?
</code></pre>
<p>Nothing fancy. Just asking the data, "Are you okay?" and listening for the answer.</p>
<ul>
<li><p>Validation gives you evidence.</p>
</li>
<li><p>Without it, all you've got is hope.</p>
</li>
</ul>
<hr />
<h2>7. To sum it up: Standards make data usable</h2>
<p>Let's go back to where we started. That full-looking spreadsheet. All those cells, all that promise, and a nagging feeling something's off.</p>
<p>Now you know why. The data was present. It just wasn't ready.</p>
<p>Standards are how it gets ready. When your data follows clear rules:</p>
<ul>
<li><p><strong>Comparisons become dependable.</strong> Apples are apples. Dates line up. Totals mean what they say.</p>
</li>
<li><p><strong>You clean less, over and over.</strong> Fix the rule once instead of fixing the mess every month.</p>
</li>
<li><p><strong>Your work can be repeated.</strong> A colleague (or future you) can run the same steps and get the same result.</p>
</li>
<li><p><strong>You trust your own analysis.</strong> And so does everyone you share it with.</p>
</li>
</ul>
<p>Cleaning data isn't merely about tidying values. It's about deciding what a value <em>should mean/represent</em>, setting a consistent rule, applying it within reason, and then checking that it worked.</p>
<p>That's the whole recipe.</p>
<p>Good data work comes down to three things: <strong>clear rules, suitable tools, and careful verification.</strong></p>
<p>Pandas is a great tool. But it's the rules and the checking that make your data something people can trust.</p>
<p>So next time you open a "full" spreadsheet, take a breath. Ask what each value should mean. Write the rule(s). Run the code. Check the result.</p>
<p>Your future self will thank you.</p>
]]></content:encoded></item></channel></rss>