28 stories
·
0 followers

Software Testing Day

3 Shares
The company tried to document how often employees were celebrating Software Testing Day, but their recordkeeping system kept mysteriously crashing.
Read the whole story
tdaitx
71 days ago
reply
Share this story
Delete

Stats for Data Science, from the Ground Up

1 Share

Data scientists love debating which skills are essential for success in the field. It makes sense: in a rapidly changing ecosystem that adopts new and powerful technologies all the time, job requirements and toolkits never stop evolving.

Statistics seem to be one major outlier, though. Data professionals of all stripes seem to agree that a solid foundation in stats and math will make your life easier regardless of your role, and can open up opportunities that would otherwise remain beyond reach.

To help you on your learning journey, we’re sharing a few of our favorite recent posts that focus on statistics for data science and machine learning. They go from the basics all the way to more specialized use cases, but they’re all accessible, beginner-friendly, and emphasize practical applications over lofty theory. Let’s dive in!

  • Stats novice? Not for long! If you’re tackling stats for the first time in your professional life—and especially if your memories of high school math inspire more dread than joy—you’re bound to appreciate Chi Nguyen’s simple explanations of basic concepts.
  • A structured approach to learning statistics. Looking for a thorough, step-by-step resource for learning stats? Adrienne Kline recently launched an excellent Statistics Bootcamp that unpacks the math behind all the data science libraries practitioners use daily. (If you’ve already discovered the first installment, linked above, parts two and three are already out!)
  • Making sense of occasionally confusing terms. For his debut TDS article, Ajay Halthor shared a lucid explanation of likelihood, and focused on the role it plays in machine learning, as well as its sometimes hard-to-grasp connection to probability, an equally crucial concept.
Photo by Alisa Anton on Unsplash
  • Putting your statistical know-how to good use. There’s always a gap between theoretical knowledge and its effective application. Mintao Wei’s recent contribution does a great job bridging it, as it walks us through the process of selecting the right statistical tests for a range of A/B testing metrics.
  • The inner workings of a powerful algorithm, explained. The bootstrap, says Christian Leschinski, “is an algorithm that allows you to determine the distribution of a test statistic without doing any theory.” It’s also one that’s been “widely overlooked.” Harnessing his deep knowledge as a statistician, Christian guides us through the magic behind the boostrap, and shows how it can help practitioners in their analyses.
  • Why it’s crucial to connect statistics to business outcomes. Cassie Kozyrkov identifies the challenges data professionals face when they bring their stats and math knowledge to work projects, and stresses the importance of data budgeting, a topic college classes rarely cover. (If you’d like to read more of Cassie’s insights—and you should!—don’t miss our brand-new Q&A with her, which touches on data career paths, the value data analysts bring to companies, and much more.)

All stats-ed out, are we? We hope not, but just in case—here are some non-statistics-related reading recommendations we think you’ll enjoy.

Your support means so much to us — thank you for reading our authors’ work; a special shoutout goes to all of you who’ve recently become Medium members.

Until the next Variable,

TDS Editors


Stats for Data Science, from the Ground Up was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read the whole story
tdaitx
689 days ago
reply
Share this story
Delete

Health Data

2 Comments and 15 Shares
Donate now to help us find a cure for causality. No one should have to suffer through events because of other events.
Read the whole story
tdaitx
788 days ago
reply
Share this story
Delete
2 public comments
synapsecracklepop
789 days ago
reply
As a medical librarian, I can confirm that this is a comic about health data.
ATL again
taddevries
790 days ago
reply
Just wow!

Debugging Flatpak applications

1 Share

Flatpak is a way to distribute applications on Linux. Its container-style approach allows applications to run across Linux distributions. This means native packages (rpm, deb, etc) are not needed and it's relatively easy to get your app to Linux users with fewer worries about distro compatibility. This makes life a lot easier for developers and is also convenient for users.

I've run popular applications like OBS Studio as flatpaks and even publish my own on Flathub, a popular hosting site for applications. Today I figured out how to debug flatpaks, which requires some extra steps that I'll share below so I don't forget them myself!

Bonus Tip: Testing local flatpaks

If you're building a flatpak of your own application it's handy to use the dir sources type in the manifest to compile your application's source code from a local directory instead of a git tag or tarball URL. This way you can make changes to the source code and test them quickly inside Flatpak.

Put something along these lines in the manifest's modules object where /home/user/my-app is you local directory with your app's source code:

{
"name": "my-app",
"sources": [
{
"type": "dir",
"path": "/home/user/my-app"
}
],
...
}

Building and installing apps with debuginfo

flatpak-builder(1) automatically creates a separate .Debug extension for your flatpak that contains your application's debuginfo. You'll need the .Debug extension if you want proper backtraces and source level debugging. At the time of writing the Flatpak documentation did not mention how to install the locally-built .Debug extension. Here is how:

$ flatpak-builder --user --force-clean --install build my.org.app.json
$ flatpak install --user --reinstall --assumeyes "$(pwd)/.flatpak-builder/cache" my.org.app.Debug

It might be a good idea to install debuginfo for the system libraries in your SDK too in case it's not already installed:

$ flatpak install org.kde.Sdk.Debug # or your runtime's SDK

Running applications for debugging

There is a flatpak(1) option that launches the application with the SDK instead of the Runtime:

$ flatpak run --user --devel my.org.app

The SDK contains development tools whereas the Runtime just has the files needed to run applications.

It can also be handy to launch a shell so you can control the launch of your app and maybe use gdb or strace:

$ flatpak run --user --devel --command=sh my.org.app
[📦 my.org.app ~]$ gdb /app/bin/my-app

Working with core dumps

If your application crashes it will dump core like any other process. However, existing ways of inspecting core dumps like coredumpctl(1) are not fully functional because the process ran inside namespaces and debuginfo is located inside flatpaks instead of the usual system-wide /usr/lib/debug location. coredumpctl(1), gdb, etc aren't Flatpak-aware and need extra help.

Use the flatpak-coredumpctl wrapper to launch gdb:

$ flatpak-coredumpctl -m <PID> my.org.app

You can get PID from the list printed by coredumpctl(1).

Conclusion

This article showed how to install locally-built .Debug extensions and inspect core dumps when using Flatpak. I hope that over time these manual steps will become unnecessary as flatpak-builder(1) and coredumpctl(1) are extended to automatically install .Debug extensions and handle Flatpak core dumps. For now it just takes a few extra commands compared to debugging regular applications.

Read the whole story
tdaitx
802 days ago
reply
Share this story
Delete

Let’s Encrypt Receives the Levchin Prize for Real-World Cryptography

1 Comment

On April 13, 2022, the Real World Crypto steering committee presented the Max Levchin Prize for Real-World Cryptography to Let’s Encrypt. The following is the speech delivered by our Executive Director, Josh Aas upon receiving the award. We’d like to thank our community for supporting us and invite you to join us in making the Internet more secure and privacy-respecting for everyone.

Thank you to the Real World Crypto steering committee and to Max Levchin for this recognition. I couldn’t be more proud of what our team has accomplished since we started working on Let’s Encrypt back in 2013.

My first temptation is to name some names, but there are so many people who have given a significant portion of their lives to this work over the years that the list would be too long. You know who you are. I hope you’re as proud as I am at this moment.

Let’s Encrypt is currently used by more than 280 million websites, issuing between two and three million certificates per day. I often think about how we got here, looking for some nugget of wisdom that might be useful to others. I’m not sure I’ve really come up with anything particularly profound, but I’m going to give you my thoughts anyway. Generally speaking: we started with a pretty good idea, built a strong team, stayed focused on what’s important, and kept ease of use in mind every step of the way.

Let’s Encrypt ultimately came from a group of people thinking about a pretty daunting challenge. The billions of people living increasingly large portions of their lives online deserved better privacy and security, but in order to do that we needed to convince hundreds of millions of websites to switch to HTTPS. Not only did we want them to make that change, we wanted most of them to make the change within the next three to five years.

Levchin Prize Trophy

We thought through a lot of options but in the end we just didn’t see any other way than to build what became Let’s Encrypt. In hindsight building Let’s Encrypt seems like it was a good and rewarding idea, but at the time it was a frustrating conclusion in many ways. It’s not an easy solution to commit to. It meant standing up a new organization, hiring at least a dozen people, understanding a lot of details about how to operate a CA, building some fairly intense technical systems, and setting all of it up to operate for decades. Many of us wanted to work on this interesting problem for a bit, solve it or at least put a big dent in it, and then move on to other interesting problems. I don’t know about you, but I certainly didn’t dream about building and operating a CA when I was younger.

It needed to be done though, so we got to work. We built a great team that initially consisted of mostly volunteers and very few staff. Over time that ratio reversed itself such that most people working on Let’s Encrypt on a daily basis are staff, but we’re fortunate to continue to have a vibrant community of volunteers who do work ranging from translating our website and providing assistance on our community forums, to maintaining the dozens (maybe hundreds?) of client software options out there.

Today there are just 11 engineers working on Let’s Encrypt, as well as a small team handling fundraising, communication, and administrative tasks. That’s not a lot of people for an organization serving hundreds of millions of websites in every country on the globe, subject to a fairly intense set of industry rules, audits, and high expectations for security and reliability. The team is preparing to serve as many as 1 billion websites. When that day comes to pass the team will be larger, but probably not much larger. Efficiency is important to us, for a couple of reasons. The first is principle - we believe it’s our obligation to do the most good we can with every dollar entrusted to us. The second reason is necessity - it’s not easy to raise money, and we need to do our best to accomplish our mission with what’s available to us.

It probably doesn’t come as a surprise to anyone here at Real World Crypto that ease of use was critical to any success we’ve had in applying cryptography more widely. Let’s Encrypt has a fair amount of internal complexity, but we expose users to as little of that as possible. Ideally it’s a fully automated and forgettable background task even to the people running servers.

The fact that Let’s Encrypt is free is a huge factor in ease of use. It isn’t even about how much money people might be willing or able to pay, but any financial transaction requirement would make it impossible to fully automate our service. At some point someone would have to get a credit card and manage payment information. That task ranges in complexity from finding your wallet to obtaining corporate approval. The existence of a payment in any amount would also greatly limit our geographic availability because of sanctions and financial logistics.

All of these factors led to the decision to form ISRG, a nonprofit entity to support Let’s Encrypt. Our ability to provide this global, reliable service is all thanks to the people and companies who believe in TLS everywhere and have supported us financially. I’m so grateful to all of our contributors for helping us.

Our service is pretty easy to use under normal circumstances, but we’re not done yet. We can be better about handling exceptional circumstances such as large revocation events. Resiliency is good. Automated, smooth resiliency is even better. That’s why I’m so excited about the ACME Renewal Info work we’re doing in the IETF now, which will go into production over the next year.

Everyone here has heard it before, but I’ll say it again because we can’t afford to let it slip our minds. Ease of use is critical for widespread adoption of real world cryptography. As we look toward the future of ISRG, our new projects will have ease of use at their core. In fact, you can learn about our newest project related to privacy-preserving measurement at two of this afternoon’s sessions! Getting ease of use right is not just about the software though. It’s a sort of pas de trois, a dance for three, between software, legal, and finance, in order to achieve a great outcome.

Thank you again. This recognition means so much to us.


Supporting Let’s Encrypt

As a nonprofit project, 100% of our funding comes from contributions from our community of users and supporters. We depend on their support in order to provide our services for the public benefit. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. If you can support us with a donation, we ask that you make an individual contribution.

Read the whole story
tdaitx
823 days ago
reply
This project changed adoption for HTTPS in the best way possible.
Share this story
Delete

AI-Generated Sleep Podcast Urges You To Imagine Pleasant Nonsense

1 Share

[Stavros Korokithakis] finds the experience of falling asleep to fairy tales soothing, and this has resulted in a fascinating project that indulges this desire by using machine learning to generate mildly incoherent fairy tales and read them aloud. The result is a fantastic sort of automated, machine-generated audible sleep aid. Even the logo is machine-generated!

The Deep Dreams Podcast is entirely machine-generated, including the logo.

The project leverages the natural language generation abilities of OpenAI’s GPT-3 to create fairytale-style content that is just coherent enough to sound natural, but not quite coherent enough to make a sensible plotline. The quasi-lucid, dreamlike result is perfect for urging listeners to imagine pleasant nonsense (thanks to Nathan W Pyle for that term) as they drift off to sleep.

We especially loved reading about the methods and challenges [Stavros] encountered while creating this project. For example, he talks about how there is more to a good-sounding narration than just pointing a text-to-speech engine at a wall of text and mashing “GO”. A good episode has things like strategic pauses, background music, and audio fades. That’s where pydub — a Python library for manipulating audio — came in handy. As for the speech, text-to-speech quality is beyond what it was even just a few years ago (and certainly leaps beyond machine-generated speech in the 80s) but it still took some work to settle on a voice that best suited the content, and the project gradually saw improvement.

Deep Dreams Podcast has a GitLab repository if you want to see the code that drives it all, and you can go to the podcast itself to give it a listen.

Read the whole story
tdaitx
840 days ago
reply
Share this story
Delete
Next Page of Stories