What a cool map of locations that LEO near me

turbofish

Learning to Fly
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
426
Reaction score
941
Location
NW Arkansas
Using data from - - - , I have made a map of all locations 75 miles near me that have been pointed out in a couple of days range [June 19-24] where radar was tagged. After creating an SSIS package, I imported all of the data so that now I have 2.3 million rows of data from just those couple of days, weeded out the junk data, put it in a nice little database and ran some queries on it. From there, I exported the results into a csv file and opened up Google Earth to get a map of all of the locations. Later on I can upload my kmz file to my server and provide a link so that you all can play. This one is for NW Arkansas since that is where I am from.

Keeping in mind that I'm at work and got a ton of projects to do, give me a state/county, I will look up the lat/lon of that county and located all of the speed traps that are in a 75 mile range. One thing that I've been playing around with is scrapping my website and starting over and from that, I can provide a place that you can easily download new locations. Trying to insert all 2.3 million rows tended to kill all of my apps that I have used for maps. If you are data inclined, I can also provide the SQL 2016 database that has compressed nicely into a 55 MB file after getting rid of the stuff I couldn't use. Attached is the CSV file.
Edit: I just noticed that the time is off in the file. Guess I didn't account for greenwich mean time-5/6 for my location
GoogleEarth.jpg


How does a data nerd play after working with huge databases all day long? By playing with more data of course.
 

Attachments

Last edited:

nighthawk

Learning to Fly
Beginner User
Joined
Jul 6, 2016
Messages
325
Reaction score
338
This is really neat, would be cool to see how you collected the data.

Might try loading the data up and play around with it, I have experience in signal processing and the like, but always been interested in big data. Never really found a data set that interested me enough to blow some time on it.

If you don't mind me asking, what kind of tools do you use to work on it? I've used R, Python, Julia and did a lot of work with Matlab.
 

turbofish

Learning to Fly
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
426
Reaction score
941
Location
NW Arkansas
I can create a csv of all of the data that includes lat/lon and a time column [normal time and epoc] but I'm a MS-SQL kind of guy. I used SSIS to loop through all of the text files and import them into a junk kind of table [all data into one column] and then created a stored procedure to dig out the data that I needed to put the data that I needed. Then I exported just the stuff that was in a 75 mile radius from me and even loaded it into PowerBI for fun.
If you like 'big data' I have other sets as well including stats on every vehicle sold since 1940 [tiny database really] and another fun project that I have been working on for fun is I have weather data for ever county, every day since 1951 until April of this year. Got some more importing to do but that one has 77 million rows of data.
I have a friend who has a daughter that just graduated from MIT in BI stuff. When I would talk to her, she would do all of the R and Python stuff and I would ask how long would it take her to process data. Most of her data sets are in the 10's of thousands of rows, mine start at 2 million and goes up to millions of rows of total data. I was amazed how much longer it would take her to process data than SQL/OLAP. Still want to learn that stuff. Once I loaded all of the GAIA spacecraft info into a database - 1.3 trillion rows of data but then I had no idea what to do with it.
 

infiniti

Advanced User
Lifetime Premium Member
Joined
Jun 8, 2014
Messages
2,441
Reaction score
2,745
Location
Ohio
Really cool. Would be amazing to have this as a data source / layer for HighwayRadar @ferius

That way, we could display a bit of a heatmap / hotspot area where speed traps typically are as we go...as long as we are wishing for the moon!
 

doubledge

PSL?
Beginner User
Joined
May 14, 2019
Messages
574
Reaction score
526
This is pretty awesome stuff. Very impressive. You should talk with @ferius and @xydrine regarding each of their software projects. This data set might be interesting to integrate into some kind of predictive model.
 

turbofish

Learning to Fly
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
426
Reaction score
941
Location
NW Arkansas
The only problem that I'm having is even when I restrict the data to just one day, the whole map lights up no matter what zip code that I use as the base. But it does show hot spots and places to watch out for
 

IPA

Gimme Shelter
Intermediate User
Lifetime Premium Member
Joined
Feb 3, 2019
Messages
709
Reaction score
880
Location
Atl
Cool work. Perhaps there’s also a way to mark breaks in the median or stretches of road without a median wall/fence for possible I/O awareness.
 

Bloovy One

110,000+ miles with JBV1!
Advanced User
Lifetime Premium Member
Joined
Feb 8, 2016
Messages
4,172
Reaction score
8,154
Location
Western NC
Here's a JBV1 generated heatmap using the last 30 days of my data. Red is Ka, Orange is K, White is Lidar, Blue is CSA alerts.

[Edited to add] Green are X-band but there are so few of those that you can't see them zoomed out like this.
 

Attachments

Last edited:

Bloovy One

110,000+ miles with JBV1!
Advanced User
Lifetime Premium Member
Joined
Feb 8, 2016
Messages
4,172
Reaction score
8,154
Location
Western NC
Not to diminish anything in this thread; but to show how it already works in JBV1:
Here's a screenshot showing both live CSAs and heat spots from recent CSAs just over the VA border. :)
 

Attachments

turbofish

Learning to Fly
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
426
Reaction score
941
Location
NW Arkansas
More maps. I loaded the data into PowerBI and then color coded based on days.
On this one, I zoomed into one area that is known for being a speed trap paradise and could see on 4 different days at around the same location, there were reports.
WithColor.jpg

This one is zoomed out a bit
WithColor2.jpg

Problems with this is I'm using PowerBI with ArcGIS plugin [both are free - PowerBI is from Microsoft] but when attempting to make a web link, the plugin doesn't work.
All done while at work! My boss thinks that a project takes 2 days but since I've done it so many times, got it done in 2 hours.

If you want to see what that looks like, I uploaded the video to my site [YouTube didn't like it]
PowerBI Map
 
Last edited:

turbofish

Learning to Fly
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
426
Reaction score
941
Location
NW Arkansas
I got to admit, JBV1 is a really cool way of visually seeing the history. --- data can be a bit much and difficult to follow. In fact, one of the reasons that I've been wanting to buy a valentine one is just for the app! Maybe one day when they have stock in inventory...
 
Last edited:

4Runner

Learning to Fly
Beginner User
Joined
Sep 19, 2019
Messages
51
Reaction score
17
BI is great!


Sent from my iPhone using Tapatalk
 

Bloovy One

110,000+ miles with JBV1!
Advanced User
Lifetime Premium Member
Joined
Feb 8, 2016
Messages
4,172
Reaction score
8,154
Location
Western NC
Did you do this from inside JBV1? I didn't know you could do that. I need to play around more with the app.
Yes. A lot has changed with JBV1 since you and I met up!
 

MakeChaiNotWar

Learning to Drive
General User
Joined
Dec 20, 2018
Messages
36
Reaction score
15
Using data from - - - , I have made a map of all locations 75 miles near me that have been pointed out in a couple of days range [June 19-24] where radar was tagged. After creating an SSIS package, I imported all of the data so that now I have 2.3 million rows of data from just those couple of days, weeded out the junk data, put it in a nice little database and ran some queries on it. From there, I exported the results into a csv file and opened up Google Earth to get a map of all of the locations. Later on I can upload my kmz file to my server and provide a link so that you all can play. This one is for NW Arkansas since that is where I am from.

Keeping in mind that I'm at work and got a ton of projects to do, give me a state/county, I will look up the lat/lon of that county and located all of the speed traps that are in a 75 mile range. One thing that I've been playing around with is scrapping my website and starting over and from that, I can provide a place that you can easily download new locations. Trying to insert all 2.3 million rows tended to kill all of my apps that I have used for maps. If you are data inclined, I can also provide the SQL 2016 database that has compressed nicely into a 55 MB file after getting rid of the stuff I couldn't use. Attached is the CSV file.
Edit: I just noticed that the time is off in the file. Guess I didn't account for greenwich mean time-5/6 for my location
View attachment 126766

How does a data nerd play after working with huge databases all day long? By playing with more data of course.
This is the most beautiful thing I've ever seen.
 

turbofish

Learning to Fly
Beginner User
Lifetime Premium Member
Joined
Nov 2, 2018
Messages
426
Reaction score
941
Location
NW Arkansas
I actually stopped doing it only because Highway Radar [ferius who I like to elect for sainthood for his work] has built in some of the same thought of marking high alert areas but it was a fun data nerd thing for me to do.
Saint Ferius - has a nice ring to it.
We also have other people here on the forum that should also be elected to sainthood like the guy who created JBVI
 

mswlogo

Software Developer
Advanced User
Premium Member
Software Developer
Joined
Nov 1, 2014
Messages
3,573
Reaction score
3,416
Location
MA
This concept has existed in V1Driver since its inception.

You can also filter by Date, Radar Band or Ka Segment.

As you Filter, Zoom and drag the map you can optionally display the distribution of those frequencies on a graph to help you tune in box muting.

49204598681_b4ec745f94_b_d.jpg
 

Discord Server

Latest threads

Latest posts

Forum statistics

Threads
81,881
Messages
1,242,921
Members
20,728
Latest member
Sam94
Top