Field Test: How Accurate is UAV Survey?

22

[Editor’s note: this blog post is part one of two. Stay tuned for the conclusion early next week.]

The buzz (sorry couldn’t resist) around UAVs is undeniable, but if I learned anything from laser scanning it’s that you shouldn’t use the manufacturer’s tech sheet to quote your capabilities.

With UAVs, this is even more of an issue as there are so many more variables to contend with. While I often think of the UAV as little more than a flying tripod, the fact is that flight control and geo-referencing options can greatly affect the outcome of projects. Then, you have issues of what type of camera to use, and that’s all before you even consider the software you intend to use for processing. However, none of this seems to deter the plethora of UAV-based service providers that call and email me each and every week looking for work.

What it comes down to is this: I don’t believe most of their claims. I simply do not understand how they can achieve the accuracies that they claim to attain. So, I decided to start running a few tests to see what was not just possible, but predictably achievable in a project setting, outside of a lab.

Before we get into the testing, a few caveats. I have very little experience with aerial mapping and fewer than 10 completed mapping projects using UAVs. Given the increase in my knowledge and the ever-increasing capabilities of UAS I am sure that whatever the results, they are the worst I will reliably achieve in my career! However, I’m hardly alone in that camp. Many new users are entering this space, either as service providers, users of the data, and/or contractors of these services. Instead of dropping in here as if I’m the expert, I thought it might prove more useful to share the learning curve as I try to learn when and where UAVs are most useful.

Lastly, I am attempting to use UAVs for mapping. While I may use them for visualization or inspection at some point, that is not the focus of this series of tests. I’m far less interested in pretty pictures than I am in accurate pictures.samb-uav-1

Testing Accuracy

As you might imagine, my first quest was to quantify accuracy and see what I can pull from the resulting data. To that end we set up a test. We had a section of road that was newly constructed and not yet open to the public. We were already contracted to map it conventionally, so our first test was to fly the area and see how accurately the UAV data matched the conventional survey. This would help us determine whether we would choose a UAV for this type of work in the future.

We also decided to laser scan a topsoil pile onsite so we could perform volumetric comparisons between the two systems. Lastly, we had access to a second UAV, and a much better camera, so we opted to cover the product pile with this unit as well for comparison.

Tech Details

We started with what I will refer to as the “basic drone.” This was the DJI Phantom 3 Professional with the standard 12 megapixel camera. We chose a pre-planned flight in a standard grid, with 80% frontal overlap and 65% side overlap. These metrics were chosen as a 5% increase over the minimum specifications suggested by the processing software, Pix4D. This was all collected at an altitude of 240 ft (73 m) relative to the elevation at the Phantom’s take off position. In the end, we took 2,051 images to cover the 309-acre (1.25 km²) site.  

For the product pile area, we also flew a Freefly Alta with a Canon 5D SR (53 megapixels) on a Freefly Mōvi gamble. As our control (scientific “control” – not survey “control”) for the product pile, we scanned it with a Leica ScanStation C10. That point cloud was georeferenced to the site survey control, as was the Phantom data. The survey control was setusing RTK to a DOT network per project specifications. The lack of accuracy in this was one of the reasons that we scanned the product pile for a more accurate comparison. I started to re-run the control for higher accuracy, but decided against it since this was to be a real-world project test–he budget would not warrant that on an actual project so with stuck with that mindset.

We processed using a combination of Pix4D and Leica Geosystems Cyclone. We did this because of the lack of a cleaning workflow in Pix4D data. I will state that I am self-taught on Pix4D, but I can’t seem to find a way to clean out vegetation and/or reduce the noise in surface data to my satisfaction. Heavy vegetation is thrown out of the data, but light vegetation as well as cars, people, tripods, etc. are all baked into the surface mesh. My work-around at this point is to export a point cloud from Pix4D so that I can clean and re-mesh it in Cyclone. I also used Cyclone for the volumetric calculations. However, the comparison to survey data for the overall project area was performed in AutoCAD Civil3D.

samb-orthophoto-uav

Data Collection

The flights went well, although it was a bit windy (steady 10-15 mph). This did not cause any visible problems with the data, but it did impact flight times by requiring more pit stops for fresh batteries compared with a calm day. The flights took place between 11am and 1pm to reduce shadows. Conditions were mostly cloudy. The DJI Phantom 3 performed as advertised along its pre-programmed flight path. The Freefly Alta was deployed in a free flight configuration, with a pilot navigating around the product pile while another technician operated the camera. Images were taken at stopping/hover points as determined by the camera operator.

Before we go too much into this system I’ll let you know that we did not get usable results. It is obviously superior tech, however, not having a location for the product pile prior to mobilization and trying the free flight method is most likely the source of our problem(s). My opinion is that we failed to maintain a consistent distance relative to the product pile, as we changed altitudes when circling the pile which caused some problems because pixels would have had wide variances in absolute size values between images within the dataset. Also, because the pile was relatively small, the onboard GPS positions all appeared to be very similar, which caused Pix4D to disregard the initial location data. I could not find a way to override this setting in the software. Either way, Pix4D did not like it at all. I am undertaking more trials toward using cameras to map 3D objects (as opposed to land forms) so I’ll get back to you if I find a way to salvage the Freefly data.

Check back next week as I’ll post up full results of the volumetric survey comparison between the C10 and the drone and cover the results in detail. However, I feel guilty bringing you this far without a preview at the very least!

Preliminary Results

Essentially, the results were comparable and for some clients they may suffice. However, in many ways they were nowhere near the accuracy levels so often quoted in the marketing materials we often receive from drone service providers. Like many collection systems, a UAS has a lot of interdependent components, all of which are capable of adding a bit of error to the overall product. Ferreting out which issues are causing me errors has lead me down a path where I’ve had to learn about things like lens aberration, rolling shutters, GSD to flight speed ratios, and that’s before you get back into traditional survey issues surrounding ground control and the like.

If there is one thing I want to leave you with at this point in the narrative, it’s that anyone operating a UAV for mapping needs to be familiar with all of these and know how to quantify their effect on his/her rig; otherwise I’d look elsewhere for a service provider.

Share.

About Author

Sam Billingsley

"Confessions of a Hired Gun" is just that, the true tales spun by a guy who's been in the field, scanning for a living, as long as anyone in the industry. What works and what doesn't work? What does the client want and what should the client want? This is a place where you'll find advice and commiseration if you're in the scanning business, and a place where you can learn about best practices and what to expect from your scanning provider if you're an engineering firm or asset owner/operator.

22 Comments

  1. Sam, high accuracy certainly is a HUGE challenge as you covered a lot of the variables. My work is in structural modeling and not survey work so it is a tangent to your article. I just want to let everybody know that the combination of LiDAR and UAV technologies CAN get relative accuracy down to sub-mm levels in a modeling scenario. We aren’t quite there yet for mm survey levels because the flight distance is further away than modeling a cell tower for instance. BTW, yes the settings in Pix4D are a challenge but after 30-40 experimental runs I did find it possible to cleanup nearly all of the noise when taking pictures 100% in a forest.

    • Sam Billingsley on

      Thanks for reading my blog! I assume you meant to say that ” UAV technologies CAN get relative accuracy down to sub-cm”? If so, is this using the combination of LiDAR/UAV you mentioned? Are you speaking of using LiDAR on the UAV or on a different platform?

      • Actually it can go sub-mm, I have done it using LiDAR and photogrammetry on towers. The LiDAR is all ground based as aerial LiDAR for this level of accuracy is not economical when compared to photogrammetry.

  2. Hi Sam, nice read. To the point and honest.
    Would you be interested in us running the same data set, through our processes and software for a side by side comparison? If so please get intouch.
    All the best,
    Chris.

  3. Sam,

    I would be happy to chat with you on getting reliable results with the drone and Pix4d. We do this all the time, and automation is the key in flight. An autopilot is absolutely necessary when performing these flights, as well as ground control if you are doing volumes that consist of anything other than monitoring earthwork.

    Best,
    Chris Klesch

    • Sam Billingsley on

      Chris, Thanks for the input. I think you’ll see in part two that we can to the same conclusion regarding autopilot. As for Ground Control Points (GCP), we did use those. I think I referred to them as “Survey Control” in the article as opposed to the GCP terminology preferred in the UAV/UAS world… …still learning all of that as well!

  4. Aloha, I’d suggest you use some GCPs around the stock pile, the GPS on the P3P isn’t super accurate thus resulting in the readings being the same (how small was the pile?) you want to capture images at least every 10 degrees as you orbit the pile while keeping the camera aimed such that you don’t have sky in the images (just ground/pile etc). You want each “key point” visible in a minimum of 5 images. You say the heavy vegetation was thrown out while in reality you failed to capture images with sufficient overlap for processing. As the trees where likely 10-20 meters in height the tops did not get captured in 5 images. UAVs are the future of mapping, but it still takes experience to get good results, and PIX4D works if you know how to use it, sounds like you would do well to spend some time reading through the online help (which is very robust). UAVs make great maps, but there is no easy button yet. I’d be happy to work with you to do a proper test

    • Sam Billingsley on

      Hi Aaron, thanks for the advice. We did use GCPs, I just referred to it as Survey Control in the article instead of the GCP terminology used in UAV/UAS circles. Sorry for the confusion!
      Your info on the heavy vegetation is quite interesting. I think you’ll see in part 2 that we determined I percentage of overlap to be a bit too low, but the “keypoint in 5 images” ratio is not one I had considered. As for Pix4D training, I have read quite a bit of their voluminous online offerings but I am looking to go to one of their training seminars this Summer as well.

  5. Thanks for the article Sam, and if you will loan me the high dollar system that failed for you in the test, I will ind a way to make it work, and let you know what the problems seem to be.

    We have been flying Phantom 3 and Phantom 4 drones for a couple of years now in our survey and mapping work, and have accurate NOAA and NGS data to compare the drone GPS to, in our work, we find the drones native GPS to be on the order of 0-5 feet from perfect, 95% or more of the time, the GPS sensors on the DJI drones are good to very good in both precision and accuracy, we have about 1000 flights under our belt at this moment.

    Having said that, it is way too early for us to trust the drone for volumetric calculations on it’s own, solid survey ground control, and a net of sparse but accurate stockpile measurements taken with GPS or Total Stations would have to be utilized to make sure the drone and it’s camera were doing their job correctly, we are in the accurate measuring business, not the accurate guessing business, after all, and long term relationships with industrial clients is predicated on our ability to accurately measure and report.

    Keep up the good work and let us know when that Freefly/Canon stuff will be in the mail to us.

    • Sam Billingsley on

      I have no doubt that the problem with our “high dollar system” was operator error! As for the volumetric accuracy I agree given the parameters, however, I do think there are some systems capable of meeting the specs for some volume projects as they can be quite loose compared to the engineering grade dimensional control stuff that many of us are accustomed to providing.
      Thanks for reading but don’t hold your breath on that shipment…

  6. Thank you very much for the tests and review of these technologies. I have been meetin g a lot of guys selling these UAVs but they failed to clearly explain how accurate the measurements would be when compared to the conventional methods.

    • Sam Billingsley on

      Thanks for reading and checking in. “Those guys” were the genesis of this article as I was tired of my clients asking why I wasn’t guaranteeing levels of accuracy that they were seeing in email blasts. I do think that the future for data collection from UASs is bright and that it will replace many methods being used today. However, it does seem to be oversold at times…

  7. Geoff Jacobs on

    Excellent write-up, Sam, on a very relevant topic in today’s marketplace. Your openness about your experience level, your observations, the tools used, and work processes is greatly appreciated. As someone who used to confer with the company’s top R&D engineers when I wrote vendor data sheets/specs for Trimble survey-grade GPS receivers and Leica scanners, I have a strong appreciation for the factors that contribute (and how much each contributes) to instrument errors and project-level errors. Keep up the good work.

  8. Interesting read Sam.
    Do you have plans to tackle the same idea with decidedly “survey grade” fixed wing UAVs such as the eBee Plus RTK and/or Mavinci?
    Full disclosure – I work as a senseFly distributor – but have found that in real world situations they are performing work that is to spec – or very close to it.
    Thanks again.

    • Sam Billingsley on

      We do plan to continue with higher grade hardware. We wanted to start with the Phantoms just due to their ubiquity within the marketplace. While I neglected to mention it in this post, we have been working with a reseller and manufacturer for tech support and best practices info moving forward. I agree that there are some UAS out there that are up to the task…

  9. Shinya Sugiura on

    I use the SfM analysis result that I performed in UAV by work in OBAYASHI CORPORATION.
    It inflects this time in the spot the scale or more.
    The precision is less than 5cm in Z direction.
    In X and Y direction, it is less than 2cm.

  10. We have done numerous tests similar to what you describe as well as those that Adam Jordan refers to. You can check out our YouTube site here for site mapping and tower mapping utilizing drones and terrestial lidar here https://www.youtube.com/user/LTC3D. We’ve flown mapping missions from 5 acres to weekly material monitoring on an active 800 acre site. Every point you make here is spot on. The biggest point of all is control, control, control. You can never have too much ground control.

  11. Keep testing it on real world environments. LiDAR is best, but good still camera frames on high contrasting elements and close range can give good 3D analysis. Using spatial filters and smoothing helps render nice mesh also. Check Martin Isenburg’s algorithms for modeling…

    • Sam Billingsley on

      I intend to as I’m confident that UAV’s are here for the duration. Thanks for the reminder about Martin’s work. I’m a big fan of LASTools but didn’t consider them here for some reason. I can only guess that I associate his work with LiDAR more than photos. I’ll have to go back and look at the modeling tools.

Leave A Reply

© Diversified Communications. All rights reserved.