Desperately seeking: Internet File transfer acceleration technology

Desperately seeking: Internet File transfer acceleration technology

Summary: What did you do Thanksgiving weekend? Me? I finally gave my new 1mpbs/8mbps (uplink/downlink) Internet service (Comcast Business Class) a stress test in hopes of proving that it will help me get the job done in terms of pumping raw DV video footage from my home office to CNET's headquarters in San Francisco as we take ZDNet's video experiment with Panasonic's P2-based AG-HVX200 to the next level.

SHARE:
TOPICS: SMBs
20

What did you do Thanksgiving weekend? Me? I finally gave my new 1mpbs/8mbps (uplink/downlink) Internet service (Comcast Business Class) a stress test in hopes of proving that it will help me get the job done in terms of pumping raw DV video footage from my home office to CNET's headquarters in San Francisco as we take ZDNet's video experiment with Panasonic's P2-based AG-HVX200 to the next level. But sadly, the test turned into disappointment when, even with the 1 mbps uplink speed (uplink speed from the home office was the challenge), it was still, for some reason, taking approximately 8 hours to upload 16 minutes of DV video (4 GB) to a shared, high-bandwidth server.

So, I hear about file transfer acceleration technology all the time. Now, I'm ready to listen. Do you have the answer to our problem? If you think you do, we'd like to put a few promising solutions to the test. What are the risks if things go badly? What are the rewards if they go well? Ink. Here.

Topic: SMBs

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

20 comments
Log in or register to join the discussion
  • Only practical solution is to use a higher compression rate

    Without sounding like "I told you so", it boils down to this. For the kind of 320x240 300-500 kbps Internet video we're posting, having source material that's 640x480 2 mbps is perfectly fine. The only way to get that kind of a stream is to use the hard drive MPEG2 DVD-format recorders.

    2 mbps video would transfer in a little more than an hour compared to the 16 hours needed for 28 mbps video.
    georgeou
    • We're trying to avoid post production...

      on this end George. It's very clear to us that we can pre-compress. But we'd rather not if we can avoid it. There's probably a solution to our problem. We just have to find it.

      db
      dberlind
      • That's my point

        With the camera you're using now, you're gonna have to convert it to a 2 mbps DV MPEG-2 lowend DVD quality stream. That's STILL WAY faster than waiting 16 hours for the file to transfer.

        With certain camera on the market, they produce a 2 mbps MPEG2 stream right off the bat for the low-quality setting. You mount the camera as an HDD and you can transfer it right there.
        georgeou
      • bigger pipe, or smaller file, i dont see the problem

        i'm flabbergasted at the question. it's very simple.

        if you have x size file and y/s size pipe, it will take x/y seconds to transfer your data. the only "accelerator" technolgies there really are involve pre-caching of the data, like in a webpage. the computer asks if it already has this information, or the proxy server skips re-downloading an image from the webserver and gives it to the requesting computer directly.

        in your situation, there is no way for the destination to already have the information you're trying to send, and hence no way to accellerate it.

        you have three options:
        compress the file making it smaller
        get a larger upload pipe
        compress the file AND get a larger upload.

        Valis
        CEO
        Valis Enterprises
        http://www.valissoft.com
        Valis Keogh
  • 4gb will take a long time no matter what. . .

    4gb transfer over even a local network takes a long time. Quick google search brought up this little calculator which shows the transfer speeds of a 10 megabit connection still being almost an hour for 4 GB. http://www.t1shopper.com/tools/calculate/downloadcalculator.shtml

    The fastest connection I've ever had for a home connection was 2megabit upload speed over DSL. If money is not an issue you can always call your local phone provider and have them install a T1 into your home.
    Also how are you transferring the file? Is this through FTP/VPN/SSH connection?
    the hero of canton
    • answer to question

      this is via FTP to shared drive space that's run by akamai.

      We are not VPNing but I did notice a slight improvement during a previous speedtest when I used SMB (Microsoft LAN protocol) over a VPN that offered some compression. So, I'm wondering if there's an accelerator that specializes in compression and decompression for video transfer. I never thought to check the VPN transferred video for lossiness... but I'm unclear whether or not it can be accomplished without loss.

      db
      dberlind
      • You can't compress video with lossless compression

        Video is already compressed. It's like trying to zip a zip file. You just end up with a larger file.
        georgeou
      • You'd have to violate fundamental CS principles to compress more

        Compressed HD video streams like the ones you are sending fundamentally cannot be compressed any more via lossless compression. If such a thing was possible and such an algorithm existed, people would be using to transmit HD video and store HD video. But such a thing is not possible without violating computer science principles.

        The only way to further compress video that's already compressed is to use lossy compression. It requires a minimum of 8 mbps to reliably compress 1080p video without excessive loss. Now if you're willing to go DV resolution (which I don't see why not since our end result is sampled down to 320x240 video), you can probably get away with transmitting 2 mbps video.

        One way or another, you're not going to avoid lossy compression unless you're getting a T3 in to your home.
        georgeou
  • Waitaminit ...

    I'm doing the math here and it looks like you're sending totally uncompressed video. Boggle.

    Well, I can understand (if not agree) with your desire to avoid lossy compression, but there are plenty of loss[b]less[/b] compression schemes for video that will save you a good bit of bandwidth.

    Unfortunately a general compression scheme like [b]bzip2[/b] won't take advantage of things like frame deltas, and I'm not aware of any video-specifig lossless compressors.
    Yagotta B. Kidding
    • No, he's sending highly compressed video

      It's a 28 mbps DV or HD stream. Both are already lossy compressed. HD is VERY compressed. Trust me on that.
      georgeou
      • It's not HD.

        It's DVC25 (25 mpbs) and when this cam does HD, it's 100 mbps.

        db
        dberlind
    • He's actually sending a highly compressed HD stream at 33 mbps

      HD takes a LOT more bandwidth than 33 mbps.
      georgeou
      • Not the case

        I'm not sure where you're getting this information George. We are not doing HD. And, when this camera does HD (a mode we're not using), it takes up 100 mbps. I don't believe there to be loss.
        dberlind
        • meh, at least george gets a response

          meh, at least george gets a response
          Valis Keogh
  • communication speed

    Seems wierd that one can have communication speeds nearing 500 mps with a local area network, but have speeds of only a few mps (if that) on the internet. Thoughts?
    tmckeon
  • pleading the "fifth"

    what about splitting the file into segments then sending them upstream from a few different IP addresses from your home.
    what are the maximum IP addresses that Comcast and SBC Yahoo allow to go into a location?
    Checking Comcast.net and "asking comcast" you get the following response;
    How many IP addresses do I receive with the Comcast High-Speed Internet Service?

    Comcast provides and authorizes one dynamically assigned IP address per residential High-Speed Internet account. You have the option of purchasing up to four additional IP addresses, for a total of five dynamic IP addresses. Please contact us for additional details.

    it's too late to research DSL, but since there are 2 pairs of phone wires into each home i *assume* you could have at least 2 IP addresses there.

    So with comcast you COULD have 5 IP addresses, segment your file in 1/5ths and upload. result being 1/5th the time

    or if time is supercruical. i suppose one could DRIVE to the office and upload
    richvball44
  • File transfer acceleration technology

    There are technologies that enable ad hock file transfer acceleration. Filecatalyst (FileCatalyst.com) by UnlimiTech software provides companies with a pure software solution that allows them to get full line speed. Deploying and using this software is as easy as setting up an FTP server and client. In fact the FileCatalyst server handles both FCP and FTP protocols. Meaning -- that you may connect to the FileCatalyst server for acceleration when required and a native FTP client when you already have full line speed. You get the best of both worlds. This product is not directed towards the home user, but rather SME?s and large multi-national companies.
    dmckeen
  • Simple math

    this makes me wonder about some of the staff at zdnet these days...

    "for some reason" took 8 hours to upload? lets do the math...

    4 gig = 4000 (roughly) MegaBytes = 32000 Megabits
    1 megabit/s upload
    to transfer 32000 megabits at 1 megabit/s = 32000 seconds

    32000 seconds = 533.33 minutes
    533.33 minutes = 8.89 hours

    um... why did it take 8 hours to upload? actually cause you were getting darn good upload speeds!!

    i'd be very happy with a solid 1meg uplink!

    Valis
    Valis Keogh
  • Exactly my sentiment

    instead of waiting 8+ hours to see the 'stress test' to finish, it would take, even doing it with a paper & pencil and recounting al the zeroes a dozen times, no longer than a fraction of a minute to come up with the best possible outcome. Not even Claude Shannon, a late theorist on capacity of communication channels, could improve on that result. The dark side of digital life is that delivering and storing digital forms takes finite time and (nearly) infinite storage.
    imrich@...
  • ??

    so what have you come up with as a solution????
    richvball44