I have not been brilliant at mathematics. I've been close to where you were, so our stories aren't too different. At the end of my story, I got my CS degree. If I can do it, you can too.
My math story starts off a bit depressing. I went all the way to pre-calc in high school, but did miserably and didn't pay attention and put in the diligent effort. I retained very little. I took a placement test and got placed quite far behind, but I felt the placement test was in error. No big deal, if the class material is easy, I can breeze right through it and fill in any gaps. I didn't have to work very hard to pass with A's. That became a bad habit. I got to pre-calculus and in order to do well, I actually had to start studying. I didn't. I passed with a C, but it's still passing. I moved onto Precalc 2. This would turn out to be a stumbling block for me. I had learned that math builds upon the previous material, sort of like a pyramid, so if your foundation is weak and you build on top of that, you are going to be miserable. With Precalc 2, I got another C. Not a great start for my adventures in calculus, so I decided I would retake it. I did, and I got another C.
Then, life kind of got in the way. I had joined the US military during the summer and 9/11 happened, and I ended up bouncing between war zones and class rooms, and being away from school for 10-12 months makes you forget things. I ended up taking Pre-calc 2 a third time as a refresher course, got around a B. I think, all in all, I ended up taking Precalc2 about 4 times. What's interesting is that, that experience made me really familiar with the material. I still can't for the life of me memorize the half angle formula or double angle formula, but I am very comfortable with sine, cosine, tangent, cosecant, unit circles, etc. This is where I learned to stop breezing by in my classes and how to actually study and learn, and how to teach myself. It was the most critical thing I learned.
I took Calc 1, got a C, didn't like it, went to war, retook the class, still got a C, and went onto pre-calc 2. Precalc 2 was actually a little easier than Calc 1, but I still got a C the first time around. I decided to take it a second time to try to improve my grade for my transcripts. I #*@!ed up. I started to get lazy. Our teacher required ALL homework to be turned in before mid terms and finals. If you didn't do your homework, you automatically failed the test. I started procrastinating on the homework. The day before the final, I started doing all of my Calc 2 homework since the mid term (around 2pm). I was "getting it" and blazing through the math as fast as I could write. Pages, upon pages, of calculus. I was writing homework until about 4am. Math class started at 8am and I had to be there in time for the final, or I would fail. I was exhausted. I decided to take a short nap, I set my alarm, and went to bed. Then, I woke up. I felt *too* rested. I didn't wake up to an alarm. Something was *very* wrong. I look at the clock. It's exactly 8:30am. If I rushed out the door, I'd get into the classroom by 9am and be an hour late. So, that's exactly what I did. I show up an hour late to take a two hour test. I felt like if I could do math as fast as I was doing it the previous night, I could still blaze through it, and maybe, if I'm lucky, get a 70%. About 30 minutes in, I realized I wasn't going to finish the test in time. My strategy changed: Rather than doing all the problems, I would only do the problems I could do the fastest. I might know how to do a problem, but if it took too long to write out the solution, I would skip it in favor of low hanging fruits. If I grab all the low hanging fruits and still have time left over, I can revisit the harder and longer problems. I turned in my test, with my homework, and... got a 55%. Ouch. I barely got a C in that class.
The thing is, I learned how to learn. I learned how to crunch. I felt, that with enough perseverance and effort, I could figure anything out. I would go onto university, and while the CS courses were quite hard and challenging, I could pass them. My secret weapon was my study habits. While others might be smarter, I even the playing field by studying harder. There were other late nights in university where I was working until 4am, but I was used to that now.
I did find a class that was harder than Calculus 2. It was "Elementary Logic", which was a 400 level course offered by my philosophy program. I loved it. The class was taught by a really old man, an obviously tenured professor, and he chose to start the class at 8am and teach in a room with the heat turned all the way up. It would be really easy to fall asleep. On the first day of class, I arrived about 10 minutes late. I walk in, and the class room is PACKED. The 30 desks were filled and about 10 more students were standing in the back taking notes. There wasn't even standing room! I ended up sitting in the window sill (they were recessed). The course material was... really hard. The textbook was really terse. You'd have to re-read the same page about 5 times just to grasp what it was saying, and another 5 times to understand it. Most logic classes are just an introduction to formal logic, this was using logic to prove that the logical proofs in formal logic were sound. We explored set theory, uncountable sets, small infinities and large infinities, russels paradox, godel, peano arithmetic, and meta logic. I remember getting stumped on the proof for the "existential specification". It was so convoluted and circular, that no matter how many times I tried to make sense of it, no matter how much I studied it, it would not click. This... was my intellectual limit it seemed. No amount of perseverance would make sense of it, and I hate rote memorization. By the middle portion of class, about 80% of the students had dropped. We were down to a small group of 12 students! Quite a contrast to the beginning of the course.
My point is, Calculus 1 & 2 may not be the hardest classes you end up taking. But, what it takes you to pass Calc 1 & 2 is going to be the same thing that carries you through the other, hard classes.
I would also say, schooling in general is actually the easy part of ones career. All of the knowledge is given to you, someone else has gone before you and found all the answers, and you are being given them by an expert. If you feel bad about the difficulty of Calculus, don't. It took mankind thousands of years to get to it, and then it was developed and refined over centuries. You are getting centuries worth of mathematical knowledge crammed into your head in a matter of months. Cut yourself some slack, but don't give up.
After you graduate and enter into the professional world, it's quite different. Technology moves fast. It's constantly changing. You may venture into areas where nobody has ever been before, doing things that have never been done. There may be no existing body of knowledge to fall back on, no experts to guide you, not even google search results to give you the answers you seek -- you may be the google results others find later. The murkiness and uncertainty may end up making you wish to go back to the black and white days of calculus classes... but really, if we're going to be honest, the same underlying universal principle is at work here: learn how to teach yourself and how to work hard. If you can master that, nothing set before you is insurmountable -- except maybe a proof of the existential specification using meta logic.