Data artist Eric Fischer built a map of the world, populated with data from geotagged Twitter photos. A blue dot means a local took a photo in that spot; a red dot means a tourist took a photo. (Fischer identified locals as anyone who took photos in one city for over a month, and tourists as anyone taking photos outside their usual city.) The maps show that tourists concentrate in certain areas, usually dense city centers, while locals spread out everywhere.
You can’t plan a trip based on this data alone, but it’s a great starting point for finding neighborhoods and streets to explore. There are clusters around most major streets, but also around walkable streets with a lot of interesting bars, restaurants, and shops. You can even spot smaller tourist traps to avoid within local neighborhoods.
The map isn’t a perfect visitors’ guide: Major parks are weirdly blank, and because the data was collected in 2010–2013, some neighborhoods are out of date. (For example, this map doesn’t show the recent tourist boom in Williamsburg.) But it still presents an overall accurate picture of where tourists and locals each congregate. And it’s fun to pick out the banks of the Thames, the path of the Staten Island Ferry, or how Hawaii’s tourists stick to the coast. Fischer also captured maps of 36 cities, including the London map above, on Flickr, where he included yellow dots for photos that could be tourists or locals: places where everyone wants to take a photo, whether they’re here every day or seeing this place for the first time ever.
In March, Matt Bratlien saw something odd in the spacious suburb of Silver Firs, north of Seattle. A six-wheeled robot with the Amazon Prime logo on its sky-blue carapace was driving up and down the sidewalks and curbs, watched by a company representative. “I was surprised, excited, and very curious,” says Bratlien, a partner at Net-Tech, an IT services company in nearby Bellevue.
Bratlien had encountered Scout, a delivery robot Amazon is testing in the area, including by ferrying real orders to customers. Here’s what he didn’t see: Countless digital clones crawling through a virtual copy of the neighborhood that Amazon created with scans of the area collected by lasers, cameras, and aircraft.
Amazon knows a lot about the world thanks to data from its vast retail business and cloud computing platform. It knows a 2-square-kilometer zone of Snohomish County in unusual detail—down to the position of weeds sprouting through the drainage grates. The company’s digital copy mirrors the position of curbstones and driveways within centimeters, and textures like the grain of asphalt within millimeters.
That synthetic suburb allows Amazon to test Scout thousands or perhaps millions of times under varying weather conditions without swarming the neighborhood with bright blue rovers until they become a nuisance. “The bots can run 24/7 in simulation,” says Sean Scott, the executive leading the project. The practice resembles how Waymo and others working on autonomous cars invest heavily in simulations to supplement miles driven on real roads.
Amazon wants Scout to help it deliver more packages, more promptly. The company recently said it would offer US members of its Prime service free next-day delivery. In January, Amazon said it was testing six of the robots somewhere in Snohomish County. It now says the fleet has grown, but declines to say exactly where they roam. County Executive Dave Somers says Amazon consulted him and the sheriff’s office prior to launching, and says he supports the project but doesn’t know exactly where Amazon tests.
Bratlien’s Facebook post geotagged Scout in Silver Firs, a community of 21,000 with curving streets branched with cul-de-sacs; another person posted a photo of Scout in the same neighborhood. Months after Amazon announced Scout was in testing, Washington Governor Jay Inslee signed a bill that regulates delivery robots, limiting their speed and weight and barring them from jaywalking.
Matt Bratlien posted this image on Facebook after encountering an Amazon robot in March.
Amazon’s mapping and simulation technology is not just a research tool. It could also help Amazon deploy the robots to new neighborhoods when they’re ready for general use, by first testing them in simulations. “We’ve built it such that we can scale up to an entire city,” Scott says. By the time Scout rolls into a new town for the first time, its control system will have likely “seen” every seam in the pavement thousands of times before.
Amazon is a relative latecomer to the young world of delivery robots. Startup Starship Technologies, founded by two creators of Skype, and competitor Marble were delivering pizza and other food orders in early 2017. Amazon’s project appears to have begun in earnest late that year when it acquired Dispatch and the startup’s three cofounders joined Amazon.
None of these companies has yet proven that delivery robots can be reliable or profitable at broad scale. Most, like Amazon, deploy their robots with human chaperones who take control in case of problems; Amazon’s attendants also unload packages at customers’ homes. Kiwi, which delivers food to students at UC Berkeley, dispatches robots unaccompanied but pays remote workers in Colombia to steer them over the internet.
Amazon’s size and deep investment in logistics set the company’s project apart. Scout joins a delivery fleet that includes 40 aircraft and 30,000 delivery vans. The company has deep experience with robotics; it employs more than 200,000 of them inside its retail operations to move shelves, load pallets, and sort packages, among other tasks.
Yet to drive safely around Silver Firs or any other neighborhood, Scout must cope with challenges not seen inside Amazon’s finely controlled warehouses. On sidewalks, Scott says, the robots are programmed to slow down and steer around people or animals—or stop if they come close. But making Scout reliably self-sufficient requires gathering as much data as possible.
Amazon built its simulated suburb in part with data from a cart similar in size to Scout that was towed behind a bicycle, capturing images using cameras and lidar, a type of 3D laser scanner used on autonomous car projects. The company filled in its map with 3D data from aircraft surveys. More recently, Amazon has experimented with a more powerful lidar too large for the bicycle cart, although Scott declines to identify how it travels.
At a glance, views from inside Amazon’s simulation can be hard to distinguish from photos of Silver Firs. Look more closely and you notice glitches like smeary foliage. All the same, Scott says his team has evidence that their fake world is accurate enough that algorithms trained on it also work on real world data.
Scout’s control system includes computer vision software that labels pixels in images from the robot’s camera as grass, sidewalk, road, or other features. A version of that software trained on 400,000 hand-labeled photos of the neighborhood scored 0.98 on an internal accuracy test with a maximum score of 1. A version trained using only images from the virtual world scored 0.94. Scott says that’s close enough to show that the simulator can help Scout understand the real world; progress doesn’t depend entirely on collecting and labeling real photos. He predicts even higher accuracy from training with a combination of real and simulated data.
One of the robot’s trickier challenges is navigating curb ramps when it needs to cross the street. Collecting data solely by repeatedly driving the ramps would be time consuming, and potentially annoying to residents. Amazon engineers can minimize that by sending virtual Scouts through curb ramps at varied trajectories from different perspectives. “It keeps us off the roads,” Scott says.
Kevin Peterson, cofounder and CEO of Marble, which is testing delivery robots in Concord, California, says his company has experimented with high-resolution simulations but found they weren’t useful enough. Instead, he says, Marble has used a technique called generative adversarial networks—also used in AI art projects—to create extra data to train its robots’ software.
Amazon could use its simulations beyond training Scout’s vision systems. A hot area in robotics research involves training control systems using a technique called reinforcement learning. Algorithms experiment many times inside a simulated world to learn a task such as how to manipulate objects through trial and error. Amazon could gain a significant advantage if it could make that approach work for its robots, says Peterson. “If had to make a bet that’s where I’d say they’re going,” he says. Scott says only that his team is exploring using learning algorithms more broadly for Scout’s control.
Some unknowns about Amazon’s robotics project are questions of economics, not engineering. Scott says his team is aiming to make Scout “100 percent” autonomous—but can also imagine robots that sometimes call home for help. “Someone could manage hundreds of bots,” he says. The robot-to-human ratio that makes Scout a practical delivery option will depend on how often it needs help, and the financial costs and payoffs of robot package hauling.
Scott says Amazon is focused on Scout for suburban deliveries for now, although Peterson of Marble says the economics are best in denser urban areas where drop offs are closer together. Amazon says Scout can travel miles at a time on a single charge, depending on the terrain and its loads.
Bratlien, who unexpectedly met Scout on the streets of Silver Firs in March, was prompted to think about how such robots might help his IT business. Net-Tech currently uses courier services to rush laptops to customers who need them right away. “This could augment that,” Bratlien says.
Moment Pro Camera, the camera app originally designed for Moment’s range of aftermarket smartphone lenses, is getting support for long-exposure photography on iOS. The feature is technically blending together a number of shorter exposures rather than creating a long single exposure like a traditional camera would, but it can be used to create a couple of different photo effects, such as light trails or motion blur. It supports exposures of predetermined lengths, or you can manually start and stop your exposure.
It’s a pretty simple implementation, but the functionality can create some neat results. I particularly liked the light trail effect, which worked well, despite the fact that I was using it in a bright room. The downside is that you really need to use a tripod if you want any parts of your image to remain sharp. No matter how hard I tried, I couldn’t hold my hand steady enough to keep the background in focus.
Once you’ve shot the image, it will save as a Live Photo to your iOS camera roll by default, but there are also options to export each image as a static JPG or as a MOV video file. You can check out how the new mode works in Moment’s video below.
Moment isn’t the only company to experiment with long-exposure photography in a camera app. Earlier this year, the team behind Halide released Spectre, an app that attempted to use AI to optimize your images. Its AI was meant to stabilize your images, so you wouldn’t get motion blur as a result of the long exposure. It was an ambitious idea, but at launch, it was a little more hit and miss in practice. The best way to get a clear shot was still to use an old-fashioned tripod.
Moment’s new long-exposure mode is releasing today on iOS where the app currently costs $5.99. The feature isn’t ready for the Android version of the app, but Moment tells me it’s looking into adding the feature at a later date.
A series of exhibitions at the Rencontres d’Arles photography festival shows how artists broke taboos about nudity, gender and sexuality.
ARLES, France — In communist Czechoslovakia, it paid to conform. But behind closed doors at Prague’s only gay club, people could be free.
In the early 1980s, Libuse Jarcovjakova, now 67, photographed the lovers, friends and strangers she met as a young woman at that underground spot, called T-Club. Many of the images have never been shown in public before, but now have a headline slot at the Rencontre d’Arles, a major annual festival of international photography in the South of France.
Ms. Jarcovjakova’s show is part of a strand of exhibitions at the festival titled “My Body is a Weapon.” Curated by Sam Stourdzé, the Rencontre d’Arles’ director, the shows celebrate the work of unheralded artists from the 1980s, who Mr. Stourdzé said “stayed alive, existed and resisted with photography.”
The portraits on display — taken in Czechoslovakia, East Germany and Spain after the Franco dictatorship — broke taboos around nudity, gender and sexuality, and asserted their subjects as individuals in societies that put the collective first.
These photographers understood that “your body is always your own, and no one can take it away,” Mr. Stourdzé said in an interview.
Ms. Jarcovjakova — whom Mr. Stourdzé calls “the Nan Goldin of Soviet Prague” — visited T-Club almost every night from 1983 to 1985, she said. She remembered the club as “a kind of family” that was a sanctuary for “outrageously camp queers” and “serious-looking men who had fled their family.”
Ms. Jarcovjakova used her camera to document it all. “I would drink vodka from the bottle and wake up with many rolls of film,” she said.
At the Academy of Performing Arts in Prague in the early 1980s, Ms. Jarcovjakova studied photography: Social realism, the austere aesthetic that promotes the valor of the working man, was the school’s official style. But she clashed with her tutors, she said. She wanted to develop a more introspective approach, and take portraits that said more about the author than the subject.
As well as documenting the community at T-Club, Ms. Jarcovjakova turned the camera on herself. She photographed herself naked, or masturbating, or lying in the bath. She also photographed her sexual partners of both sexes: naked, undressing and in bed.
This came with risks. Same-sex activity was decriminalized in Czechoslovakia in 1962, but homophobia was rife and the charge of gross indecency was routinely used to arrest gay, lesbian and bisexual people, according to Ms. Jarcovjakova. Exhibiting images that explored the desires of a young, bisexual woman would have provoked a reaction from the authorities.
So Ms. Jarcovjakova hid her work away, “in a cardboard box in the back of my cupboard,” she said. The photographs remained there until four years ago, when she met the Prague-based curator Lucie Cerná.
Ms. Cerná arranged for Mr. Stourdzé to view a selection of the portraits at the Paris Photo fair, and Mr. Stourdzé almost immediately offered Ms. Jarcovjakova a major show at the festival in Arles.
The varying ways artists reacted to surveillance by the state is a thread that runs through “My Body is a Weapon.” Another exhibition in the strand, “Restless Bodies,” is a group-show of 16 artists from East Berlin who worked clandestinely in East Germany, known as the German Democratic Republic.
The socialist East Germany was a totalitarian state in which artists had to be trained by and registered with the authorities. Sonia Voss, the curator of “Restless Bodies,” said in an interview that many of the artists in the show taught at schools or worked for the state-run news media. All of them were under surveillance by the Stasi, the secret police, she said.
In their portraits, which were not intended for public display, the photographers used their own bodies to communicate a sense of self and to express their interior lives, Ms. Voss said. “They spent a lot of time undressing and dressing up,” she added.
One of those artists, Christiane Eisler, took austere portraits of the underground punk scene in the city of Leipzig. Another, Rudolf Schäfer, secretly shot carefully framed and exquisitely lit portraits of the recently deceased in the morgue of the Charité Hospital in East Berlin.
Gabriele Stötzer, a photographer with work in “Restless Bodies,” said in an interview in Arles that she had spent time in an East German prison after taking part in a demonstration at college. After her release, she was monitored by the Stasi, she said.
In the show, Ms. Stötzer exhibits a series of portraits she took in 1985 of a young man who poses for the camera in tights, high heels and makeup. “He was a transvestite,” Ms. Stötzer said. This was taboo in East Germany, Ms. Voss said, so taking such portraits was an act of defiance.
After the fall of the Berlin Wall, when the Stasi’s archives were made available to the public, Ms. Stötzer discovered that the subject of the portraits had been informing on her to the secret police. “The Stasi took advantage of anyone who was different,” she said. “They knew his difference made him susceptible, vulnerable. They used it against him.”
Fifteen years before the German Democratic Republic was swept away, freedom had come to Spain, as seen in “La Movida,” an exhibition at the Rencontre d’Arles that looks back on the countercultural explosion in Madrid as the country transitioned to democracy after Franco’s death.
“La Movida” shows the emergence of a sexually liberated, genre-fluid youth movement that lit up the city’s streets. Influenced by punk and New Wave, that moment in Spain’s history was “a free for all” in which “everything was valid,” according to the exhibition’s curator, Irene de Mendoza.
“A new generation used their body to exhibit and determine their own identity” by dressing outlandishly, taking drugs and exploring sexuality, Ms. de Mendoza said.
Paul Moorhouse, a curator of photography at the National Portrait Gallery in London, said in a telephone interview that bodies remained a preoccupation for many photographers in the 21st century, and that questions of identity politics and diversity were key themes for artists nowadays.
But they were “concerned with representations of the body; artists in the 1980s were more interested in explorations of the body,” he added.
Mr. Stourdzé, the festival director, said that “today, a lot of photographers don’t need to resist.” However, he added, “they’re working in a similar way” to the artists in “My Body Is a Weapon.”
Mr. Stourdzé said that elsewhere at the festival in Arles, many exhibiting artists owed a debt to the pioneering work of the 1980s dissidents. The Chinese photographer’s Pixy Liao’s study of patriarchy in China, which features the photographer’s own nude body, was an example, he said.
“These works are rooted into a photographic past that grew out of resistance,” Mr. Stourdzé said.
Ms. Jarcovjakova said that if she were in the position of a young photographer today, her advice for herself would be to “Keep going, keep doing it, don’t stop.’”
“Because that’s how I found my freedom,” she said. “In myself.”
YouTube is currently under investigation by the Federal Trade Commission following complaints that the platform improperly collected data from young users. It’s unclear how much data this might be, but there’s reason to believe it could be a lot. For many kids, YouTube has replaced television; depending on how parents use online platforms, children could begin to amass data even before birth.
Sophie Allaert is a French attorney at law; Mélina Cardinal-Bradette works in human rights law; and Elif Sert is a research affiliate at the Berkman Klein Center for Internet & Society at Harvard University. They are graduates of the UC Berkeley School of Law.
Eighty-one percent of the world’s children and 92 percent of US children now have an online presence before they turn 2. In addition, 95 percent of US teens report having (or having access to) a smartphone. And 45 percent of those teens are online on a near-constant basis, an average of nine hours each day.
Some preeminent tech figures, such as Facebook CEO Mark Zuckerberg and Apple CEO Tim Cook, have asserted that “data ownership” is the answer to this massive online footprint, in which users control their own data and decide when to allow corporations or governments to use it.
Though this idea may sound appealing, it is not a sufficient tool in protecting individuals—especially children—from the pervasive effects of an uncontrollable online identity.
First, ownership makes no sense when the subject isn’t the creator of the content. Indeed, a person cannot remove content published about them by someone else. During their earliest years, kids’ digital identities are shaped by other individuals, most likely their parents. That means a massive amount of public information about them might be generated before they are able to understand what it means to give consent.
Furthermore, data can be aggregated. Regardless of whether a person uses online services, some decisions will still be made without their control—even without their knowledge—through inference algorithms.
Imagine that a child avoids having a digital footprint—that neither this child’s parents nor the child herself has ever used or posted anything online. Institutions can still use data about other youngsters who fall into similar categories (such as those with the same zip code or those who go to the same school) to make inferences about the child. To put it simply, even if a child is somehow shielded from a premature online identity, his or her life will still be influenced by the online presence of similar children.
The practice of data collection could have far-reaching consequences for children’s fundamental rights. The Convention on the Rights of the Child, the most ratified human rights treaty ever, protects children as individuals. But modern technology raises new questions: Will children self-censor themselves on the internet because they don’t know how their data will be used? How is access to information limited when social media platforms use algorithms to display personalized and targeted content? We don’t know what ramifications widespread data collection could have on future generations of kids.
To protect children’s fundamental rights, we need a new data protection framework: one based on how the data is used, not who owns it.
There are already some provisions in place. The Children’s Online Privacy Protection Act of 1998 (COPPA) requires operators of websites and online services to obtain parents’ explicit consent before collecting the personal information of children under 13. Children—or at least their parents—own their personal information and can decide when to share it with third parties. COPPA also attempted to regulate how online providers can market to children. For instance, a website operator cannot require a child to disclose personal information in order to participate in a game. Nonetheless, even with parents’ consent, companies still end up storing, collecting, and sharing children’s information.
Thus, the concept of data ownership is not sufficient to protect children’s privacy rights. We need broader regulation on how data is used, as well as a legal framework that explicitly protects our fundamental civil, political, and socioeconomic rights online.
Data collection—and its use—should be limited. (The Data Care Act, a bill introduced by US senator Brian Schatz last December, proposes that data must be used carefully, loyally, and confidentially.) The framework should apply to all relevant stakeholders, including governments, companies, and individuals. It should define technical standards that prioritize privacy and establish uniform practices for online platform employees, such as the engineers building these systems. Finally, non-compliance from companies should result in sanctions or other actionable consequences.
Faced with the increased usage of artificial intelligence and the growing capabilities of data processing, change is urgently needed. Under the Generation AI Initiative, UNICEF Innovation and the Human Rights Center at UC Berkeley School of Law recently published a Memorandum on AI and Child Rights. The research sheds light on how new technologies might affect children’s freedom of expression, as well as their rights not to be subjected to discrimination or abuse.
Past generations were able to grow up without a digital record of their past. This generation, and the ones to come, will be held accountable to their inescapable online identities. How current regulations respond to this shift is a fundamental question of our time.
WIRED Opinion publishes pieces written by outside contributors and represents a wide range of viewpoints. Read more opinions here. Submit an op-ed at firstname.lastname@example.org
To begin with, a programming note: While you’re reading this in its usual time slot, it was compiled before the July Fourth holiday, so—unfortunately—there’s not a collection of tweets about President Trump’s reportedly expensive Independence Day celebration. Apologies for that. Nonetheless, there’s still a lot to catch up on. Last week was one that uncovered a secret Facebook group where immigration officials joke about migrant deaths, even as more and more people die in US custody and conditions at border facilities turn out to be worse than believed. It was also the week where the Trump administration let go of its quest to have a question about citizenship on the 2020 US census, and everyone continued to fall in love with the US women’s soccer team. As to what else people were talking about, there’s also all of this.
One Small Step for Man
What Happened: President Trump completed a genuine first by not only visiting the DMZ on the North Korean border, but stepping into North Korea itself.
What Really Happened: The week started with an unexpected moment of political history on the world stage, as President Trump—in Japan for meetings—made an extraordinaryinvitationviaTwitter.
On Sunday morning, news started to circulate that the meeting might actually happen.
History was already being made simply by Trump’s presence at the border, but then there was this stunning turn of events.
Was it all a publicity stunt? Was it a moment of diplomatic history? It remains unclear, but either way, it was certainlyaverybigstorythathadeveryonetalking. But the question was left hanging: What actually happened here..? Because the first takes were particularly grim.
Well, it’s not as if the president has legitimized the North Korean leadership, undoing decades of international policy or anything… (I mean, OK, he didinvite Kim Jong-un to the US.)
So what actually lay behind this historic moment? Did the president simply want a win in his column, and thought this was a way to get one? Was he lured in by the promise of a photo-ready historic moment? Or, perhaps, he simply saw a way to claim bragging rights that, as it turned out, weren’treallyathing.
The confusion surrounding this moment really feels as if folks got a strange insight into how the diplomacy sausage is made, in a sense.
The Takeaway: Still, at least negotiations are going well.
The New White House Press Secretary Has Arrived
What Happened: This past week Stephanie Grisham stepped into her new position as the White House press secretary. According to some reports, she came out swinging, and not in the direction that many would have expected.
What Really Happened: For all the conversation about the photo op potential of the Trump/Kim Jong-un meeting and visit to North Korea, it should be noted that the international trip was the first outing for new White House press secretary Stephanie Grisham. Her debut did not go unnoticed, for an entirely unlikely reason.
What press secretary for the President of the United States hasn’t had their first big moment described as an “all out brawl,” am I right? It’s the traditional hazing for everyone in the job! As Grisham herself startedtobecomethestory, details continued to trickle in via Twitter.
It’s hard not to imagine how awkward this must have been for Grisham, who had somehow managed to overshadow her new boss on what was essentially her first day on the job (technically, her third, I believe), despite him actually doing something of significant historical importance. While people may never know how she felt about all of this, they do know how the press felt, because they talked about it.
Indeed, she earned op-eds filled with praise for her actions, as if she was somehow an avatar of a new age where the Trump administration wasn’t going to attack the press itself at any potential provocation. It was, surreally, the ideal introduction for Grisham in her new position—even for those who’d rather the White House press secretary tame the news media, rather than defend them.
As should only be expected, the video of the event didn’t live up to everyone’s expectations, leading to the birth of Grisham Brawl Truthers.
The Takeaway: For anyone wondering how the president took the whole affair, the fact that it was almost immediately used as fodder by the official re-election campaign Twitter account might tell you everything you need to know.
The Origins of Homelessness, According to Trump
What Happened: President Trump shared some rather interesting ideas about the origins of America’s homeless population.
What Really Happened: Last week, President Trump sat down for an interview with Fox News’ Tucker Carlson during which the host compared the “clean” streets of Japanese cities with the “filth” found in American cities. During that exchange, Trump shared some rather surprising views on how long homelessness has been an issue in America.
Yes, he stated that it all started two years ago. The president’s comments did not go unnoticed, to say the least.
The reality of the situation is entirely at odds with how the president described it, and CNN’s fact-checker found no evidence to back up Trump’s claims.
The Takeaway: The reality of the situation is entirely at odds with how the president described it, and CNN’s fact-checker found no evidence to back up Trump’s claims.
What Happened: It shouldn’t be surprising that Vice President Mike Pence is in the White House, unless, of course, he was supposed to be somewhere else entirely.
What Really Happened: As if things weren’t surreal enough, last week Mike Pence kind of … disappeared. On Tuesday, there was a very odd, very public case of confusion about the veep’s whereabouts—or, to be more specific, the reasons why he wasn’t where he was supposed to be.
It all started with what seemed to be a relatively straightforward emergency, if that doesn’t sound like a contradiction in terms.
OK, so, so far so … nerve-wracking. But things got weird soon after, when Pence’s press secretary decided to deny everything.
So, what actually happened? Despite the best attempts of Pence’s staff, it’s something where the mysterysoonbecamethestory. Well, we say “best attempts,” but the explanations given were more like working through the process of elimination for all interested parties.
Apparently, the public find everything ou t… eventually.
Who doesn’t love it when politics comes with built-in cliffhangers?
The Takeaway: What are the odds that this turns out to be significantly less terrifying than it currently feels?
In any conventional silicon-based solar cell, there is an absolute limit on overall efficiency, based partly on the fact that each photon of light can only knock loose a single electron, even if that photon carried twice the energy needed to do so. But now, researchers have demonstrated a method for getting high-energy photons striking silicon to kick out two electrons instead of one, opening the door for a new kind of solar cell with greater efficiency than was thought possible.
While conventional silicon cells have an absolute theoretical maximum efficiency of about 29.1 percent conversion of solar energy, the new approach, developed over the last several years by researchers at MIT and elsewhere, could bust through that limit, potentially adding several percentage points to that maximum output. The results are described today in the journal Nature, in a paper by graduate student Markus Einzinger, professor of chemistry Moungi Bawendi, professor of electrical engineering and computer science Marc Baldo, and eight others at MIT and at Princeton University.
The basic concept behind this new technology has been known for decades, and the first demonstration that the principle could work was carried out by some members of this team six years ago. But actually translating the method into a full, operational silicon solar cell took years of hard work, Baldo says.
That initial demonstration “was a good test platform” to show that the idea could work, explains Daniel Congreve PhD ’15, an alumnus now at the Rowland Institute at Harvard, who was the lead author in that prior report and is a co-author of the new paper. Now, with the new results, “we’ve done what we set out to do” in that project, he says.
The original study demonstrated the production of two electrons from one photon, but it did so in an organic photovoltaic cell, which is less efficient than a silicon solar cell. It turned out that transferring the two electrons from a top collecting layer made of tetracene into the silicon cell “was not straightforward,” Baldo says. Troy Van Voorhis, a professor of chemistry at MIT who was part of that original team, points out that the concept was first proposed back in the 1970s, and says wryly that turning that idea into a practical device “only took 40 years.”
The key to splitting the energy of one photon into two electrons lies in a class of materials that possess “excited states” called excitons, Baldo says: In these excitonic materials, “these packets of energy propagate around like the electrons in a circuit,” but with quite different properties than electrons. “You can use them to change energy — you can cut them in half, you can combine them.” In this case, they were going through a process called singlet exciton fission, which is how the light’s energy gets split into two separate, independently moving packets of energy. The material first absorbs a photon, forming an exciton that rapidly undergoes fission into two excited states, each with half the energy of the original state.
But the tricky part was then coupling that energy over into the silicon, a material that is not excitonic. This coupling had never been accomplished before.
As an intermediate step, the team tried coupling the energy from the excitonic layer into a material called quantum dots. “They’re still excitonic, but they’re inorganic,” Baldo says. “That worked; it worked like a charm,” he says. By understanding the mechanism taking place in that material, he says, “we had no reason to think that silicon wouldn’t work.”
What that work showed, Van Voorhis says, is that the key to these energy transfers lies in the very surface of the material, not in its bulk. “So it was clear that the surface chemistry on silicon was going to be important. That was what was going to determine what kinds of surface states there were.” That focus on the surface chemistry may have been what allowed this team to succeed where others had not, he suggests.
The key was in a thin intermediate layer. “It turns out this tiny, tiny strip of material at the interface between these two systems [the silicon solar cell and the tetracene layer with its excitonic properties] ended up defining everything. It’s why other researchers couldn’t get this process to work, and why we finally did.” It was Einzinger “who finally cracked that nut,” he says, by using a layer of a material called hafnium oxynitride.
The layer is only a few atoms thick, or just 8 angstroms (ten-billionths of a meter), but it acted as a “nice bridge” for the excited states, Baldo says. That finally made it possible for the single high-energy photons to trigger the release of two electrons inside the silicon cell. That produces a doubling of the amount of energy produced by a given amount of sunlight in the blue and green part of the spectrum. Overall, that could produce an increase in the power produced by the solar cell — from a theoretical maximum of 29.1 percent, up to a maximum of about 35 percent.
Actual silicon cells are not yet at their maximum, and neither is the new material, so more development needs to be done, but the crucial step of coupling the two materials efficiently has now been proven. “We still need to optimize the silicon cells for this process,” Baldo says. For one thing, with the new system those cells can be thinner than current versions. Work also needs to be done on stabilizing the materials for durability. Overall, commercial applications are probably still a few years off, the team says.
Other approaches to improving the efficiency of solar cells tend to involve adding another kind of cell, such as a perovskite layer, over the silicon. Baldo says “they’re building one cell on top of another. Fundamentally, we’re making one cell — we’re kind of turbocharging the silicon cell. We’re adding more current into the silicon, as opposed to making two cells.”
The researchers have measured one special property of hafnium oxynitride that helps it transfer the excitonic energy. “We know that hafnium oxynitride generates additional charge at the interface, which reduces losses by a process called electric field passivation. If we can establish better control over this phenomenon, efficiencies may climb even higher.” Einzinger says. So far, no other material they’ve tested can match its properties.
The research was supported as part of the MIT Center for Excitonics, funded by the U.S. Department of Energy.
With .NET Core 3.0 Preview 6 out the door, we thought it would be useful to take a brief look at the history of our infrastructure systems and the significant improvements that have been made in the last year or so.
This post will be interesting if you are interested in build infrastructure or want a behind-the-scenes look at how we build a product as big as .NET Core. It doesn’t describe new features or sample code that you should use in your next application. Please tell us if you like these types of posts. We have a few more like this planned, but would appreciate knowing if you find this type of information helpful.
A little history
Begun over 3 years ago now, the .NET Core project was a significant departure from traditional Microsoft projects.
Developed publicly on GitHub
Composed of isolated git repositories that integrate together vs. a monolithic repository.
Targets many platforms
Its components may ship in more than one ‘vehicle’ (e.g. Roslyn ships as a component of Visual Studio as well as the SDK)
Our early infrastructure decisions were made around necessity and expediency. We used Jenkins for GitHub PR and CI validation because it supported cross-platform OSS development. Our official builds lived in Azure DevOps (called VSTS at the time) and TeamCity (used by ASP.NET Core), where signing and other critical shipping infrastructure exists. We integrated repositories together using a combination of manually updating package dependency versions and somewhat automated GitHub PRs. Teams independently built what tooling they needed to do packaging, layout, localization and all the rest of the usual tasks that show up in big development projects. While not ideal, on some level this worked well enough in the early days. As the project grew from .NET Core 1.0 and 1.1 into 2.0 and beyond we wanted to invest in a more integrated stack, faster shipping cadences and easier servicing. We wanted to produce a new SDK with the latest runtime multiple times per day. And we wanted all of this without reducing the development velocity of the isolated repositories.
Many of the infrastructure challenges .NET Core faces stem from the isolated, distributed nature of the repository structure. Although it’s varied quite a bit over the years, the product is made up of anywhere from 20-30 independent git repositories (ASP.NET Core had many more until recently). On one hand, having many independent development silos tends to make development in those silos very efficient; a developer can iterate very quickly in the libraries without much worry about the rest of the stack. On the other hand, it makes innovation and integration of the overall project much less efficient. Some examples:
If we need to roll out new signing or packaging features, doing so across so many independent repos that use different tools is very costly.
Moving changes across the stack is slow and costly. Fixes and features in repositories ‘low’ in the stack (e.g. corefx libraries) may not be seen in the SDK (the ‘top’ of the stack) for several days. If we make a fix in dotnet/corefx, that change must be built and the new version flowed into any up-stack components that reference it (e.g. dotnet/core-setup and ASP.NET Core), where it will be tested, committed and built. Those new components will then need to flow those new outputs further up the stack, and so on and so forth until the head is reached.
In all of these cases, there is chance for failure at many levels, further slowing down the process. As .NET Core 3.0 planning began in earnest, it became clear that we could we could not create a release of the scope that we wanted without significant changes in our infrastructure.
A three-pronged approach
We developed a three-pronged approach to ease our pain:
Shared Tooling (aka Arcade) – Invest in shared tooling across our repositories.
System Consolidation (Azure DevOps) – Move off of Jenkins and into Azure DevOps for our GitHub CI. Move our official builds from classic VSTS-era processes onto modern config-as-code.
Automated Dependency Flow and Discovery (Maestro) – Explicitly track inter-repo dependencies and automatically update them on a fast cadence.
Prior to .NET Core 3.0, there were 3-5 different tooling implementations scattered throughout various repositories, depending on how you counted.
While in this world each team gets to customize their tooling and only build exactly what they need, it does have some significant downsides:
Developers move between repositories less efficiently
Example: When a developer moves from dotnet/corefx into dotnet/core-sdk, the ‘language’ of the repository is different. What does she type to build and test? Where do the logs get placed? If she needs to add a new project to the repo, how is this done?
Each required feature gets built N times
Example: .NET Core produces tons of NuGet packages. While there is some variation (e.g. shared runtime packages like Microsoft.NETCore.App produced out of dotnet/core-setup are built differently than ‘normal’ packages like Microsoft.AspNet.WebApi.Client), the steps to produce them are fairly similar. Unfortunately, as repositories diverge their layout, project structure, etc. it generates differences in how these packaging tasks need to be implemented. How does a repository define what packages should be generated, what goes in those packages, their metadata, and so on. Without shared tooling, it is often easier for a team to just implement another packaging task rather than reuse another. This is of course a strain on resources.
With Arcade, we endeavored to bring all our repos under a common layout, repository ‘language’, and set of tasks where possible. This is not without its pitfalls. Any kind of shared tooling ends up solving a bit of a ‘Goldilocks’ problem. If the shared tooling is too prescriptive, then the kind of customization required within a project of any significant size becomes difficult, and updating that tooling becomes tough. It’s easy to break a repository with new updates. BuildTools suffered from this. The repositories that used it became so tightly coupled to it that it was not only unusable for other repositories, but making any changes in buildtools often broke consumers in unexpected ways. If shared tooling is not prescriptive enough, then repositories tend to diverge in their usage of the tooling, and rolling out updates often requires lots of work in each individual repository. At that point, why have shared tooling in the first place?
Arcade actually tries to go with both approaches at the same time. It defines a common repository ‘language’ as set of scripts (see eng/common), a common repository layout, and common set of build targets rolled out as an MSBuild SDK. Repositories that choose to fully adopt Arcade have predictable behavior, making changes easy to roll out across repositories. Repositories that do not wish to do so can pick and choose from a variety of MSBuild task packages that provide basic functionality, like signing and packaging, that tend to look the same across all repositories. As we roll out changes to these tasks, we try our best to avoid breaking changes.
Let’s take a look at the primary features that Arcade provides and how they integrate into our larger infrastructure.
Common build task packages – These are a basic layer of MSBuild tasks which can either be utilized independently or as part of the Arcade SDK. They are “pay for play” (hence the name ‘Arcade’). They provide a common set of functionality that is needed in most .NET Core repositories:
Common repo targets and behaviors – These are provided as part of an MSBuild SDK called the “Arcade SDK”. By utilizing it, repositories opt-in to the default Arcade build behaviors, project and artifact layout, etc.
Common repository ‘language’ – A set of common script files that are synchronized between all the Arcade repositories using dependency flow (more on that later). These script files introduce a common ‘language’ for repositories that have adopted Arcade. Moving between these repositories becomes more seamless for developers. Moreover, because these scripts are synced between repositories, rolling out new changes to the original copies located in the Arcade repo can quickly introduce new features or behavior into repositories that have fully adopted the shared tooling.
Shared Azure DevOps job and step templates – While the scripts that define the common repository ‘language’ are primarily targeted towards interfacing with humans, Arcade also has a set of Azure DevOps job and step templates that allow for Arcade repositories to interface with the Azure DevOps CI systems. Like the common build task packages, the step templates form a base layer that can be utilized by almost every repository (e.g. to send build telemetry). The job templates form more complete units, allowing repositories to worry less about the details of their CI process.
Moving to Azure DevOps
As noted above, the larger team used a combination of CI systems through the 2.2 release:
AppVeyor and Travis for ASP.NET Core’s GitHub PRs
TeamCity for ASP.NET’s official builds
Jenkins for the rest of .NET Core’s GitHub PRs and rolling validation.
Classic (non-YAML) Azure DevOps workflows for all non-ASP.NET Core official builds.
A lot of differentiation was simply from necessity. Azure DevOps did not support public GitHub PR/CI validation, so ASP.NET Core turned to AppVeyor and Travis to fill the gap while .NET Core invested in Jenkins. Classic Azure DevOps did not have a lot of support for build orchestration, so the ASP.NET Core team turned to TeamCity while the .NET Core team built a tool called PipeBuild on top of Azure DevOps to help out. All of this divergence was very expensive, even in some non-obvious ways:
While Jenkins is flexible, maintaining a large (~6000-8000 jobs), stable installation is a serious undertaking.
Building our own orchestration on top of classic Azure DevOps required a lot of compromises. The checked in pipeline job descriptions were not really human-readable (they were just exported json descriptions of manually created build definitions), secrets management was ugly, and they quickly became over-parameterized as we attempted to deal with the wide variance in build requirements.
When official build vs. nightly validation vs. PR validation processes are defined in different systems, sharing logic becomes difficult. Developers must take additional care when making process changes because and breaks are common. We defined Jenkins PR jobs in a special script file, TeamCity had lots of manually configured jobs, AppVeyor and Travis used their own yaml formats, and Azure DevOps had the obscure custom system we built on top of it. It was easy to make a change to build logic in a PR and break the official CI build. To mitigate this, we did work to keep as much logic in scripting common to official CI and PR builds, but invariably differences creep in over time. Some variance, like in build environments, is basically impossible to entirely remove.
Practices for making changes to workflows varied wildly and were often difficult to understand. What a developer learned about Jenkins’s netci.groovy files for updating PR logic did not translate over to the PipeBuild json files for official CI builds. As a result, knowledge of the systems was typically isolated to a few team members, which is less than ideal in large organizations.
When Azure DevOps began to roll out YAML based build pipelines and support for public GitHub projects as .NET Core 3.0 began to get underway, we recognized we had a unique opportunity. With this new support, we could move all our existing workflows out of the separate systems and into modern Azure DevOps and also make some changes to how we deal with official CI vs. PR workflows. We started with the following rough outline of the effort:
Keep all our logic in code, in GitHub. Use the YAML pipelines everywhere.
Have a public and private project.
The public project will run all the public CI via GitHub repos and PRs as we always have
The private project will run official CI be the home of any private changes we need to make, in repositories matching the public GitHub repositories
Only the private project will have access to restricted resources.
Share the same YAML between official CI and PR builds. Use template expressions to differentiate between the public and private project where behavior must diverge, or resources only available in the private project would be accessed. While this often makes the overall YAML definition a little messier, it means that:
The likelihood of a build break when making a process change is lower.
A developer only really needs to change one set of places to change official CI and PR process.
Build up Azure DevOps templates for common tasks to keep duplication of boilerplate YAML to a minimum, and enable rollout of updates (e.g. telemetry) easy using dependency flow.
As of now, all of the primary .NET Core 3.0 repositories are on Azure DevOps for their public PRs and official CI. A good example pipeline is the official build/PR pipeline for dotnet/arcade itself.
Maestro and Dependency Flow
The final piece of the .NET Core 3.0 infrastructure puzzle is what we call dependency flow. This is not a unique concept to .NET Core. Unless they are entirely self-contained, most software projects contain some kind of versioned reference to other software. In .NET Core, these are commonly represented as NuGet packages. When we want new features or fixes that libraries have shipped, we pull those new updates by updating the referenced version numbers in our projects. Of course, these packages may also have versioned references to other packages, those other packages may have more references, so on and so forth. This creates a graph. Changes flow through the graph as each repository pulls new versions of their input dependencies.
A Complex Graph
The primary development life-cycle (what developers regularly work on) of most software projects typically involves a small number of inter-related repositories. Input dependencies are typically stable and updates are sparse. When they do need to change, it’s usually a manual operation. A developer evaluates the available versions of the input package, chooses an appropriate one, and commits the update. This is not the case in .NET Core. The need for components to be independent, ship on different cadences and have efficient inner-loops development experiences has led to a fairly large number of repositories with a large amount of inter-dependency. The inter-dependencies also form a fairly deep graph:
We also expect that new outputs will flow quickly through this graph so that the end product can be validated as often as possible. For instance, we expect the latest bits of ASP.NET Core or the .NET Core Runtime to express themselves in the SDK as often as possible. This essentially means updating dependencies in each repository on a regular, fast cadence. In a graph of sufficient size, like .NET Core has, this quickly becomes an impossible task to do manually. A software project of this size might go about solving this is a number of ways:
Auto-floating input versions – In this model, dotnet/core-sdk might reference the Microsoft.NETCore.App produced out of dotnet/core-setup by allowing NuGet to float to the latest prerelease version. While this works, it suffers from major drawbacks. Builds become non-deterministic. Checking out an older git SHA and building will not necessarily use the same inputs or produce the same outputs. Reproducing bugs becomes difficult. A bad commit in dotnet/core-setup can break any repository pulling in its outputs, outside of PR and CI checks. Orchestration of builds becomes a major undertaking, because separate machines in a build may restore packages at different times, yielding different inputs. All of these problems are ‘solvable’, but require huge investment and unnecessary complication of the infrastructure.
‘Composed’ build – In this model, the entire graph is built all at once in isolation, in dependency order, using the latest git SHAs from each of the input repositories. The outputs from each stage of the build are fed into the next stage. A repository effectively has its input dependency version numbers overwritten by its input stages. At the end of a successful build, the outputs are published and all the repositories update their input dependencies to match what was just built. This is a bit of an improvement over auto-floating version numbers in that individual repository builds aren’t automatically broken by bad check-ins in other repos, but it still has major drawbacks. Breaking changes are almost impossible to flow efficiently between repositories, and reproducing failures is still problematic because the source in a repository often doesn’t match what was actually built (since input versions were overwritten outside of source control).
Automated dependency flow – In this model, external infrastructure is used to automatically update dependencies in a deterministic, validated fashion between repositories. Repositories explicitly declare their input dependencies and associated versions in source, and ‘subscribe’ to updates from other repositories. When new builds are produced, the system finds matching subscriptions, updates any of the declared input dependencies, and opens a PR with the changes. This method improves reproducibility, the ability to flow breaking changes, and allows a repository owner to have control over how updates are done. On the downside, it can be significantly slower than either of the other two methods. A change can only flow from the bottom of the stack to the top as fast as the total sum of the PR and Official CI times in each repository along the flow path.
.NET Core has tried all 3 methods. We floated versions early on in the 1.x cycle, had some level of automated dependency flow in 2.0 and went to a composed build for 2.1 and 2.2. With 3.0 we decided to invest heavily in automated dependency flow and abandon the other methods. We wanted to improve over our former 2.0 infrastructure in some significant ways:
Ease traceability of what is actually in the product – At any given repository, it’s generally possible to determine what versions of what components are being used as inputs, but almost always hard to find out where those components were built, what git SHAs they came from, what their input dependencies were, etc.
Reduce required human interaction – Most dependency updates are mundane. Auto-merge the update PRs as they pass validation to speed up flow.
Keep dependency flow information separate from repository state – Repositories should only contain information about the current state of their node in the dependency graph. They should not contain information regarding transformation, like when updates should be taken, what sources they pull from, etc.
Flow dependencies based on ‘intent’, not branch – Because .NET Core is made up of quite a few semi-autonomous teams with different branching philosophies, different component ship cadences, etc. do not use branch as a proxy for intent. Teams should define what new dependencies they pull into their repositories based on the purpose of those inputs, not where they came from. Furthermore, the purpose of those inputs should be declared by those teams producing those inputs.
‘Intent’ should be deferred from the time of build – To improve flexibility, avoid assigning the intent of a build until after the build is done, allowing for multiple intentions to be declared. At the time of build, the outputs are just a bucket of bits built at some git SHA. Just like running a release pipeline on the outputs of an Azure DevOps build essentially assigns a purpose for the outputs, assigning an intent to a build in the dependency flow system begins the process of flowing dependencies based on intent.
With these goals in mind, we created a service called Maestro++ and a tool called ‘darc’ to handle our dependency flow. Maestro++ handles the data and automated movement of dependencies, while darc provides a human interface for Maestro++ as well as a window into the overall product dependency state. Dependency flow is based around 4 primary concepts: dependency information, builds, channels and subscriptions.
Builds, Channels, and Subscriptions
Dependency information – In each repository, there is a declaration of the input dependencies of the repository along with source information about those input dependencies in the eng/Version.Details. Reading this file, then transitively following the repository+sha combinations for each input dependency yields the product dependency graph.
Builds – A build is just the Maestro++ view on an Azure DevOps build. A build identifies the repository+sha, overall version number and the full set of assets and their locations that were produced from the build (e.g. NuGet packages, zip files, installers, etc.).
Channels – A channel represents intent. It may be useful to think of a channel as a cross repository branch. Builds can be assigned to one or more channels to assign intent to the outputs. Channels can be associated with one or more release pipelines. Assignment of a build to a channel activates the release pipeline and causes publishing to happen. The asset locations of the build are updated based on release publishing activities.
Subscriptions – A subscription represents transform. It maps the outputs of a build placed on a specific channel onto another repository’s branch, with additional information about when those transforms should take place.
These concepts are designed so that repository owners do not need global knowledge of the stack or other teams’ processes in order to participate in dependency flow. They basically just need to know three things:
The intent (if any) of the builds that they do, so that channels may be assigned.
Their input dependencies and what repositories they are produced from.
What channels they wish to update those dependencies from.
As an example, let’s say I own the dotnet/core-setup repository. I know that my master branch produces bits for day to day .NET Core 3.0 development. I want to assign new builds to the pre-declared ‘.NET Core 3.0 Dev’ channel. I also know that I have several dotnet/coreclr and dotnet/corefx package inputs. I don’t need to know how they were produced, or from what branch. All I need to know is that I want the newest dotnet/coreclr inputs from the ‘.NET Core 3.0 Dev’ channel on a daily basis, and the newest dotnet/corefx inputs from the ‘.NET Core 3.0 Dev’ channel every time they appear.
First, I onboard by adding an eng/Version.Details file. I then use the ‘darc’ tool to ensure that every new build of my repository on the master branch is assigned by default to the ‘.NET Core 3.0 Dev’ channel. Next, I set up subscriptions to pull inputs from .NET Core 3.0 Dev for builds of dotnet/corefx, dotnet/coreclr, dotnet/standard, etc. These subscriptions have a cadence and auto-merge policy (e.g. weekly or every build).
As the trigger for each subscription is activated, Maestro++ updates files (eng/Version.Details.xml, eng/Versions.props, and a few others) in the core-setup repo based on the declared dependencies intersected with the newly produced outputs. It opens a PR, and once the configured checks are satisfied, will automatically merge the PR.
This in turn generates a new build of core-setup on the master branch. Upon completion, automatic assignment of the build to the ‘.NET Core 3.0 Dev’ channel is started. The ‘.NET Core 3.0 Dev’ channel has an associated release pipeline which pushes the build’s output artifacts (e.g. packages and symbol files) to a set of target locations. Since this channel is intended for day to day public dev builds, packages and symbols are pushed to various public locations. Upon release pipeline completion, channel assignment is finalized and any subscriptions that activate on this event are fired. As more components are added, we build up a full flow graph representing all of the automatic flow between repositories.
Coherency and Incoherency
The increased visibility into the state of .NET Core’s dependency graph highlighted an existing question: What happens when multiple versions of the same component are referenced at various nodes in the graph? Each node in .NET Core’s dependency graph may flow dependencies to more than one other node. For instance, the Microsoft.NETCore.App dependency, produced out of dotnet/core-setup, flows to dotnet/toolset, dotnet/core-sdk, aspnet/extensions and a number of other places. Updates of this dependency will be committed at different rates in each of those places, due to variations in pull request validation time, need for reaction to breaking changes, and desired subscription update frequencies. As those repositories then flow elsewhere and eventually coalesce under dotnet/core-sdk, there may be a number of different versions of Microsoft.NETCore.App that have been transitively referenced throughout the graph. This is called incoherency. When only a single version of each product dependency is referenced throughout the dependency graph, the graph is coherent. We always strive to ship a coherent product if possible.
What kinds of problems of does incoherency cause? Incoherency represents a possible error state. For an example let’s take a look at Microsoft.NETCore.App. This package represents a specific API surface area. While multiple versions of Microsoft.NETCore.App may be referenced in the repository dependency graph, the SDK ships with just one. This runtime must satisfy all of the demands of the transitively referenced components (e.g. WinForms and WPF) that may execute on that runtime. If the runtime does not satisfy those demands (e.g. breaking API change), failures may occur. In an incoherent graph, because all repositories have not ingested the same version of Microsoft.NETCore.App, there is a possibility that a breaking change has been missed.
Does this mean that incoherency is always an error state? No. For example, let’s say that the the incoherency of Microsoft.NETCore.App in the graph only represents a single change in coreclr, a single non-breaking JIT bug fix. There would technically be no need to ingest the new Microsoft.NETCore.App at each point in the graph. Simply shipping the same components against the new runtime will suffice.
If incoherency only matters occasionally, why do we strive to ship a coherent product? Because determining when incoherency does not matter is hard. It is easier to simply ship with coherency as the desired state than attempt to understand any semantic effects differences between incoherent components will have on the completed product. It can be done, but on a build to build basis it is time intensive and prone to error. Enforcing coherency as the default state is safer.
Dependency Flow Goodies
All this automation and tracking has a ton of advantages that become apparent as the repository graph gets bigger. It opens up a lot of possibility to solve real problems we have on a day to day basis. While we have just begun to explore this area, the system can begin to answer interesting questions and handling scenarios like:
What ‘real’ changes happened between git SHA A and SHA B of dotnet/core-sdk? – By building up a full dependency graph by walking the Version.Details.xml files, I can identify the non-dependency changes change happened in the graph.
How long will it take for a fix to appear in the product? – By combining the repository flow graph and per-repository telemetry, we can estimate how long it will take to move a fix from repo A to repo B in the graph. This is especially valuable late in a release, as it helps us make a more accurate cost/benefit estimation when looking at whether to take specific changes. For example: Do we have enough time to flow this fix and complete our scenario testing?
What are the locations of all assets produced by a build of core-sdk and all of its input builds?
In servicing releases, we want to take specific fixes but hold off on others. Channels could be placed into modes where a specific fix is allowed to flow automatically through the graph, but others are blocked or require approval.
As .NET Core 3.0 winds down, we’re looking for new areas to improve. While planning is still in the (very) early stages, we expect investments in a some key areas:
Reduce the time to turn a fix into a shippable, coherent product – The number of hops in our dependency graph is significant. This allows repositories a lot of autonomy in their processes, but increases our end to end ‘build’ time as each hop requires a commit and official build. We’d like to significantly reduce that end-to-end time.
Improving our infrastructure telemetry – If we can better track where we fail, what our resource usage looks like, what our dependency state looks like, etc. we can better determine where our investments need to be to ship a better product. In .NET Core 3.0 we took some steps in this direction but we have a ways to go.
We’ve evolved our infrastructure quite a bit over the years. From Jenkins to Azure DevOps, from manual dependency flow to Maestro++, and from many tooling implementations to one, the changes we’ve made to ship .NET Core 3.0 are a huge step forward. We’ve set ourselves up to develop and ship a more exciting product more reliably than ever before.
Some 33,000 years ago, a man was violently clubbed to death by a left-handed attacker wielding a club or similar object. That’s the conclusion of an international team of scientists, who published the results of their forensic analysis in a recent paper in PLOS ONE.
The so-called Cioclovina calvaria is a fossilized skull around 33,000 years old, discovered in a cave in South Transylvania in 1941 during a mining operation. That makes it one of the earliest fossilized human remains yet known, so naturally it’s been studied extensively by scientists interested in learning more about the Upper Paleolithic period, which started around 40,000 to 45,000 years, and marks the major dispersal of modern humans in Europe.
“The Cioclovina individual is particularly important, as it is one of the earliest and relatively complete skulls of modern Europeans from the Upper Paleolithic period,” co-author Katerina Harvati of Eberhard Karls University of Tubingen in Germany told Live Science. “Human remains from this period are very rare and often very fragmentary.”
It was eventually determined be the skull of a male, and earlier researchers had noted that there were two small, healed scars on the frontal bone—evidence of some kind of trauma that occurred sufficiently prior to death to give the damaged area time to remodel (antemortem).
That is largely undisputed, but there was also a large fracture on the right parietal lobe. And scholars disagreed on whether this was evidence for blunt force trauma and possibly the man’s cause of death (a perimortem injury). Alternatively, the fracture could have developed after death (postmortem). The fact that it isn’t described in the original 1942 publication describing the then-newly discovered skull lends some support to this latter view, but not enough to settle the debate once and for all.
I’m a firm believer in the power of a live performance. A television broadcast or DVD doesn’t capture the same thing as a theatrical production or a concert. You gotta be there.
But what about when you can’t? What recourse is there when you’re in love with an artist or performer who you can’t physically interact with for any number of reasons?
I’ve thought about this for decades from a few perspectives: as a former full-time music critic; as a frequent chronicler of how information is presented and exchanged online; and perhaps most of all, as a music fan who had one freaking band slip through his hands.
I will never see my favorite songwriter in concert, right in front of me, reacting to my cheers and enthusiasm. But if any performer was going to vanish just as I teetered into my concert-going years, at least this one had some surprises for me five, 10, even 20 years later, all just a few mouse-clicks away.
Hexbyte – Tech News – Ars Technica | “I was little, I didn’t know $#!*”
If you’ve heard of the musician in question, Mark Sandman, you likely travel in some select music circles. Sandman was best known as the lead singer and bassist for the Boston-area rock trio Morphine, who rose to mid-level, college-radio fame in the mid-’90s. Theirs was a unique sound: just bass, drums, saxophone, and vocals, swirled together in a style that Sandman dubbed “low rock.” Saxophonist Dana Colley played his baritone sax like a guitar, while Sandman’s unique slide-bass style favored thudding chords and far-from-subtle grooves.
I found out they existed the same way a few thousand people from my generation did: by a quick-hit report on MTV News. The cable channel would occasionally air five-minute vignettes about up-and-coming bands, usually when they’d signed to mid-level record-contract deals. Morphine’s “You Hear It First” snippet wasn’t well preserved; I’ve struggled to find it online over the past 20-plus years, but I can pin it to a specific era and a specific friend who I watched it with. One of those seared-in-the-brain childhood memories.
But at that age, if it wasn’t in a BMG or Columbia House catalog (“10 albums for one penny!”), I pretty much couldn’t buy a CD in question. So my real appreciation of the band was held up a few years, until I started going to the local used CD store to spend whatever part-time job money I hadn’t spent at Software At Cost Plus 10% (an actual store in Dallas for a few years). I flipped through the CD store’s “new arrivals” bin, where the good stuff usually hid before getting alphabetically sorted by staffers. One time, I found a Morphine motherlode. I bought it all.
Were I typing this article at a music-centric site, I might regale you with a lengthy, overwrought description—one of exhilaration at the band’s crunching, unique-sounding riffs, another of my love for Sandman’s straight-to-the-gut lyrics, and a third about how foolish I immediately felt for not looking up their albums sooner. Instead, I’ll just embed a song.
I knew this was outside the normal rock-radio realm of the time, and I also quickly figured out that not every band with a CD at my local store had quite “made it.” That was OK for me. I didn’t fit in at my school. It was cool to have a band to reinforce that feeling—something weird, edgy, catchy, and poetic that didn’t require me to dress like a metalhead or a goth.
But as I hinted earlier, my increasing fandom didn’t go according to plan. My hopes of eventually seeing Morphine in concert fell apart 20 years ago this week. On July 3, 1999, Sandman died at the age of 47, shortly after suffering a heart attack in the middle of a concert in Rome. No foul play or drugs were detected; Sandman had a history of congenital heart failure in his family, and he’d brushed aside troubling symptoms earlier that day. The rest of that 1999 summer tour was, unsurprisingly, canceled. I had just turned 18 and would’ve finally been able to see them at age-restricted venues.
Hexbyte – Tech News – Ars Technica | “11 years later, still don’t know any better”
I know a lot about Sandman’s death. I also know a lot about the album that was about to launch when he died, about the other bandmates’ efforts to create a tribute concert series in the wake of the tragedy, about a pair of documentaries about Sandman’s legacy, and more.
I wasn’t nearly as encyclopedic about Morphine at the time. In hindsight, I realize how numb I was to Sandman’s death. Other musicians I really liked in my teens had come and gone: Kurt Cobain, Layne Staley, and Shannon Hoon, just off the top of my head. And my fascination with music was only beginning to blossom; barely two months after Sandman’s death, I had moved to Austin for college, where I became a live-music addict. Between cheap concerts and free file-sharing downloads, I was distracted.
My mourning period for Sandman didn’t really land for some time. I’ve come to realize that’s typical for grief, in terms of life events reminding us of the people and things we miss. But my first blush with this about Sandman came from a surprisingly geeky source: Soulseek, one of the many peer-to-peer file-sharing apps that lit